gavindiaz/poly-maker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Getting there

Browse Source
This commit is contained in:
Daniel Sapkota
2026-07-05 17:50:04 -04:00
parent bfacef317b
commit c7b686116e
92 changed files with 8062 additions and 7964 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+24 -6
View File
@@ -1,7 +1,25 @@
# Polymarket Authentication
PK=your_private_key_here
BROWSER_ADDRESS=your_wallet_address_here
# polymaker secrets. Copy to .env and fill in. NEVER commit .env.
#
# IMPORTANT: use the SAME wallet you use in the Polymarket browser UI, and make
# sure it has traded at least once through the UI so allowances are set.
# Google Sheets (for data_updater)
SPREADSHEET_URL=https://docs.google.com/spreadsheets/d/1Kt6yGY7CZpB75cLJJAdWo7LSp9Oz7pjqfuVWwgtn7Ns/edit?gid=97507557#gid=97507557
#replace with YOUR url
# Private key of the signing wallet (EOA that controls the funder).
PK=
# The funder = the smart-contract wallet that actually holds your pUSD and
# positions (NOT your signing EOA/MetaMask address above). Polymarket's UI labels
# this inconsistently (may show as "deposit" or "developer" address) — the one
# that holds the funds is the funder. `polymaker doctor` reads the balance so you
# can confirm you picked the right one. Set signature_type in config/config.toml
# to match how the account was made (new deposit wallets = 3).
BROWSER_ADDRESS=
# Optional: override the default public Polygon RPC with your own (Alchemy/Infura).
# POLYGON_RPC=
# Optional: webhook (Discord/Telegram/ntfy) POST URL for critical alerts.
# ALERT_WEBHOOK_URL=
# Optional: route outbound traffic through an SSH tunnel / proxy, e.g. to test
# from a colocated box. Standard env var; httpx and web3 pick it up automatically.
# ALL_PROXY=socks5://127.0.0.1:1080
.gitignore
+10
View File
@@ -467,3 +467,13 @@ cython_debug/
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
# polymaker v2
.venv/
state.db
state.db-*
journal/
logs/
positions/
data/
*.whl
.python-version
+1 -1
View File
@@ -1 +1 @@
3.9.18
3.12
README.md
+124 -145
View File
@@ -1,159 +1,138 @@
# Poly-Maker
# poly-maker (v2)
A market making bot for Polymarket prediction markets. This bot automates the process of providing liquidity to markets on Polymarket by maintaining orders on both sides of the book with configurable parameters. A summary of my experience running this bot is available [here](https://x.com/defiance_cr/status/1906774862254800934)
A maker-only market-making bot for **Polymarket CLOB V2**, focused on political
markets. Single async process, local-file config (no Google Sheets), typed and
tested. This is a ground-up rewrite of v1 — see [`docs/scoping/`](docs/scoping/00-overview.md)
for the design and [`docs/scoping/06-migration-plan.md`](docs/scoping/06-migration-plan.md)
for the plan. The original v1 code has been removed; see the git history (branch
`main`) if you need to reference it.
## Overview
Poly-Maker is a comprehensive solution for automated market making on Polymarket. It includes:
- Real-time order book monitoring via WebSockets
- Position management with risk controls
- Customizable trade parameters fetched from Google Sheets
- Automated position merging functionality
- Sophisticated spread and price management
## Structure
The repository consists of several interconnected modules:
- `poly_data`: Core data management and market making logic
- `poly_merger`: Utility for merging positions (based on open-source Polymarket code)
- `poly_stats`: Account statistics tracking
- `poly_utils`: Shared utility functions
- `data_updater`: Separate module for collecting market information
## Requirements
- Python 3.9.10 or higher
- Node.js (for poly_merger)
- Google Sheets API credentials
- Polymarket account and API credentials
## Installation
This project uses UV for fast, reliable package management.
### Install UV
```bash
# macOS/Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
# Or with pip
pip install uv
```
### Install Dependencies
```bash
# Install all dependencies
uv sync
# Install with development dependencies (black, pytest)
uv sync --extra dev
```
### Quick Start
```bash
# Run the market maker (recommended)
uv run python main.py
# Update market data
uv run python update_markets.py
# Update statistics
uv run python update_stats.py
```
### Setup Steps
#### 1. Clone the repository
```bash
git clone https://github.com/yourusername/poly-maker.git
cd poly-maker
```
#### 2. Install Python dependencies
```bash
uv sync
```
#### 3. Install Node.js dependencies for the merger
```bash
cd poly_merger
npm install
cd ..
```
#### 4. Set up environment variables
```bash
cp .env.example .env
```
#### 5. Configure your credentials in `.env`
Edit the `.env` file with your credentials:
- `PK`: Your private key for Polymarket
- `BROWSER_ADDRESS`: Your wallet address
**Important:** Make sure your wallet has done at least one trade through the UI so that the permissions are proper.
#### 6. Set up Google Sheets integration
- Create a Google Service Account and download credentials to the main directory
- Copy the [sample Google Sheet](https://docs.google.com/spreadsheets/d/1Kt6yGY7CZpB75cLJJAdWo7LSp9Oz7pjqfuVWwgtn7Ns/edit?gid=1884499063#gid=1884499063)
- Add your Google service account to the sheet with edit permissions
- Update `SPREADSHEET_URL` in your `.env` file
#### 7. Update market data
Run the market data updater to fetch all available markets:
```bash
uv run python update_markets.py
```
This should run continuously in the background (preferably on a different IP than your trading bot).
- Add markets you want to trade to the "Selected Markets" sheet
- Select markets from the "Volatility Markets" sheet
- Configure parameters in the "Hyperparameters" sheet (default parameters that worked well in November are included)
#### 8. Start the market making bot
```bash
uv run python main.py
```
> [!WARNING]
> In today's market, this bot is not profitable and will lose money. Use it as a reference implementation for building your own market making strategies, not as a ready-to-deploy solution. Given the increased competition on Polymarket, I don't see a point in playing with this unless you're willing to dedicate a significant amount of time.
> Market making on Polymarket is competitive and can lose money. This is a
> reference implementation and a research harness, not a guaranteed-profitable
> product. Test in `--paper` mode first; go live with small size.
## Configuration
## What it does
The bot is configured via a Google Spreadsheet with several worksheets:
- Discovers political markets via the **Gamma API** (seconds, not the v1 hour-long
crawl) and ranks them by reward + rebate income vs. volatility/spread risk.
- Maintains a live order book per token from the **market WebSocket**.
- Quotes **maker-only** (every order is post-only): a fair-value + inventory-skew
strategy that posts BUY-YES and BUY-NO as the canonical two-sided quote, with
live volatility/toxicity estimation and a regime machine that pulls quotes
during news events. See [`docs/scoping/04-strategy.md`](docs/scoping/04-strategy.md).
- Reconciles a target quote set against live orders with churn tolerances; runs
the exchange **heartbeat** dead-man switch; enforces risk caps and a daily-loss
kill switch.
- Config, market selection, and state are **local files + SQLite**. An operator
with the repo, a `.env`, and a funded wallet is a complete deployment.
- **Selected Markets**: Markets you want to trade
- **All Markets**: Database of all markets on Polymarket
- **Hyperparameters**: Configuration parameters for the trading logic
## Install
Uses [uv](https://docs.astral.sh/uv/) and Python 3.12+.
## Poly Merger
```bash
uv sync --extra dev # install deps + dev tools
uv run polymaker --help
```
The `poly_merger` module is a particularly powerful utility that handles position merging on Polymarket. It's built on open-source Polymarket code and provides a smooth way to consolidate positions, reducing gas fees and improving capital efficiency.
## Configure
## Important Notes
```bash
cp .env.example .env # then edit: PK + BROWSER_ADDRESS (same wallet as the UI)
```
- This code interacts with real markets and can potentially lose real money
- Test thoroughly with small amounts before deploying with significant capital
- The `data_updater` is technically a separate repository but is included here for convenience
### Which wallet address?
Polymarket's current **deposit-wallet** architecture (rolled out ~June 2026)
shows several addresses in the UI, and the labels are inconsistent. What matters:
- **`BROWSER_ADDRESS` = the funder** — the *smart-contract wallet that actually
holds your pUSD and positions*. Depending on the account it may be shown as the
"deposit" or "developer" address; the reliable test is which one holds the
money. `polymaker doctor` reads the balance so you can confirm.
- **`PK` = the private key of your signer** — the EOA (e.g. your MetaMask account)
that *owns/controls* that wallet. This is a **different** address than the
funder, and that's correct: your key signs on behalf of the wallet that holds
the funds. (A "deployer" / factory address, if shown, is Polymarket's shared
contract — ignore it.)
Set `signature_type` in `config/config.toml` to match how the account was made:
`3` = POLY_1271 deposit wallet (current default), `2` = older browser-wallet
Gnosis Safe, `1` = email/magic proxy, `0` = plain EOA. A wrong type fails loudly
(`polymaker doctor` / `livetest` report it). Approvals must have been granted
from the funding wallet — trading once in the UI does this. See
[`docs/scoping/03-api-layer.md`](docs/scoping/03-api-layer.md) §1/§9.
Everything else is TOML under [`config/`](config/):
- `config.toml` — wallet/engine/risk/execution settings
- `strategy.toml` — named parameter profiles (`political-longdated`, `political-hot`)
- `markets.toml` — the trade list (populated via the CLI below)
## Use
```bash
# 1. discover + rank political markets (writes to state.db)
uv run polymaker scan
uv run polymaker markets
# 2. add markets to the trade list
uv run polymaker markets-add <slug> --profile political-longdated
# 3. dry run: full pipeline against the live feed, no orders posted
uv run polymaker run --paper
# 4. preflight the wallet before going live
uv run polymaker doctor
# 5. one safe live round-trip (~$5 post-only order, placed deep and cancelled)
uv run polymaker livetest
# 6. go live
uv run polymaker run
# ops
uv run polymaker status # positions / open orders
uv run polymaker cancel-all # panic button
```
## Architecture
```
market WS ─▶ OrderBook ─▶ (wake) ─▶ Quoter ─▶ strategy (pure) ─▶ reconcile ─▶ ExecutionGateway
user WS ─▶ StateStore RiskManager ┘ (post-only, heartbeat)
Gamma ─▶ Catalog/scanner ─▶ SQLite periodic REST reconcile ┘
```
The strategy layer is a pure function `(book, inventory, params, clock) →
TargetQuotes` — deterministic and unit-tested. The engine owns all I/O and
state around it. Full component map in
[`docs/scoping/02-architecture.md`](docs/scoping/02-architecture.md).
## Develop
```bash
uv run pytest # unit suite (offline)
POLYMAKER_LIVE=1 uv run pytest tests/test_live_marketdata.py # live WS integration
uv run ruff check src tests # lint
uv run mypy src # types (strict)
```
## Status / roadmap
Implemented: config, catalog/scanner, order book + analytics, strategy (FV,
vol/toxicity, regime, quoting), state store + lifecycle, execution gateway +
reconciler + heartbeat, market/user websockets, risk manager, merger (EOA path),
engine, CLI, paper mode, journal capture. 83 tests; the market-data path is
verified live.
Spike-gated (need the live wallet — see [`docs/scoping/03-api-layer.md`](docs/scoping/03-api-layer.md) §9):
V2 order signing with a Safe/proxy wallet (`livetest` is the probe), pUSD
wrap/allowance mechanics, exact user-WS field names, native Safe merge execution.
Deferred (scoped, not built): replay backtester over captured journals, external
data feeds (polls/news/cross-venue — [`docs/scoping/07-future-external-data.md`](docs/scoping/07-future-external-data.md)).
## License
config/config.toml
+48
View File
@@ -0,0 +1,48 @@
# polymaker global config. Secrets live in .env, never here.
# See docs/scoping/05-config-and-ops.md.
[wallet]
# chain: Polygon mainnet
chain_id = 137
# signature_type — pick based on how your Polymarket account was created:
# 0 = EOA (funds held directly by your key; no proxy)
# 1 = email/magic proxy wallet
# 2 = older browser-wallet Gnosis Safe proxy
# 3 = POLY_1271 deposit wallet (the CURRENT architecture, ~June 2026+)
# If your UI shows a "deposit address", you almost certainly have a deposit
# wallet -> use 3. In every proxy case (1/2/3), BROWSER_ADDRESS is the DEPOSIT
# address (the funder that holds your pUSD), NOT your signing EOA and NOT the
# deployer/factory. Known SDK auth bugs live here (docs 03 §9) — `polymaker
# doctor` / `livetest` will tell you if you picked the wrong type.
signature_type = 3
clob_host = "https://clob.polymarket.com"
gamma_host = "https://gamma-api.polymarket.com"
data_api_host = "https://data-api.polymarket.com"
polygon_rpc = "https://polygon-rpc.com"
[engine]
debounce_ms = 200 # min gap between quote recomputes per market
reconcile_interval_s = 30 # REST drift reconciliation cadence
catalog_refresh_s = 900 # market catalog rescan cadence (15 min)
heartbeat = true # exchange dead-man switch
heartbeat_interval_s = 5
journal = true # append raw WS/orders to journal/ for backtest
loop = "uvloop" # "uvloop" | "asyncio"
[risk]
max_total_exposure_usdc = 5000.0 # sum of |position notional| + open buy notional
max_event_group_loss_usdc = 1000.0 # neg-risk group worst-case loss cap
max_market_notional_usdc = 800.0 # per-market position+orders notional cap
daily_loss_kill_usdc = 250.0 # realized daily loss -> halt new quotes
ws_stale_halt_s = 10.0 # no book updates for this long -> halt market
max_order_error_rate = 0.25 # rolling order-post error fraction -> halt
[execution]
rate_budget_fraction = 0.25 # fraction of documented API limits we allow
post_only = true # every quote is post-only (maker-only mandate)
max_orders_per_batch = 15 # exchange batch cap
[paths]
db = "state.db"
journal_dir = "journal"
log_dir = "logs"
config/markets.toml
+11
View File
@@ -0,0 +1,11 @@
# The trade list (replaces the v1 Selected Markets sheet).
# Each entry names a market by slug OR condition_id and a strategy profile.
# `polymaker markets add <slug>` appends here; edit freely, hot-reloaded live.
# Start empty — populate with `polymaker scan` then `polymaker markets`.
# Example (disabled) entry:
# [[markets]]
# slug = "will-the-democrats-win-the-2028-us-presidential-election"
# profile = "political-longdated"
# enabled = false
# q_max_usdc = 800 # optional per-market override of the profile value
config/strategy.toml
+66
View File
@@ -0,0 +1,66 @@
# Named strategy parameter profiles (replaces the v1 Hyperparameters sheet).
# markets.toml maps each market to one of these profiles.
# See docs/scoping/04-strategy.md §8.
[profiles.political-longdated]
# --- fair value ---
micro_levels = 3 # depth levels used for microprice
flow_ewma_halflife_s = 120 # signed-flow EWMA half-life
# --- spread / skew ---
gamma = 0.5 # inventory risk aversion (skew strength)
delta_min_ticks = 2 # minimum half-spread, in ticks
c_vol = 1.2 # half-spread added per unit short-horizon vol
c_tox = 2.0 # half-spread added per unit toxicity score
# --- vol horizons (seconds) ---
vol_short_halflife_s = 10
vol_long_halflife_s = 900
# --- sizing / inventory ---
base_size_usdc = 50.0 # notional per quote
q_max_usdc = 500.0 # hard inventory cap (notional)
q_soft_frac = 0.6 # soft cap as fraction of q_max
layers = 2 # price levels per side
layer_step_ticks = 2 # tick gap between layers
# --- placement / churn ---
reprice_ticks = 2 # only reprice if target moves this many ticks
resize_frac = 0.15 # or if size drifts this fraction
min_edge_ticks = 1 # never quote inside (FV +/- this) * tick
# --- regime ---
event_cooloff_s = 60
event_jump_ticks = 8 # FV jump over debounce that flags EVENT
event_sweep_levels = 3 # levels consumed in one print that flags EVENT
trend_flow_z = 1.5 # flow z-score that flags TRENDING
# --- lifecycle ---
end_date_taper_days = 7
reduce_only_hours = 24
halt_before_hours = 2
# --- exits ---
exit_urgency_s = 900 # time to walk exit from FV+delta to best-bid+tick
merge_min_size = 20.0 # min(YES,NO) shares to trigger a merge
[profiles.political-hot]
# tighter, defensive profile for high-volatility / event-prone markets
micro_levels = 3
flow_ewma_halflife_s = 60
gamma = 0.9
delta_min_ticks = 3
c_vol = 1.8
c_tox = 3.0
vol_short_halflife_s = 8
vol_long_halflife_s = 600
base_size_usdc = 30.0
q_max_usdc = 250.0
q_soft_frac = 0.5
layers = 2
layer_step_ticks = 3
reprice_ticks = 2
resize_frac = 0.15
min_edge_ticks = 1
event_cooloff_s = 120
event_jump_ticks = 6
event_sweep_levels = 2
trend_flow_z = 1.2
end_date_taper_days = 7
reduce_only_hours = 24
halt_before_hours = 2
exit_urgency_s = 600
merge_min_size = 20.0
data_updater/erc20ABI.json
-279
View File
@@ -1,279 +0,0 @@
[
{
"constant": true,
"inputs": [],
"name": "name",
"outputs": [
{
"name": "",
"type": "string"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "guy",
"type": "address"
},
{
"name": "wad",
"type": "uint256"
}
],
"name": "approve",
"outputs": [
{
"name": "",
"type": "bool"
}
],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "totalSupply",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "src",
"type": "address"
},
{
"name": "dst",
"type": "address"
},
{
"name": "wad",
"type": "uint256"
}
],
"name": "transferFrom",
"outputs": [
{
"name": "",
"type": "bool"
}
],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "wad",
"type": "uint256"
}
],
"name": "withdraw",
"outputs": [],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "decimals",
"outputs": [
{
"name": "",
"type": "uint8"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "",
"type": "address"
}
],
"name": "balanceOf",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": true,
"inputs": [],
"name": "symbol",
"outputs": [
{
"name": "",
"type": "string"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"constant": false,
"inputs": [
{
"name": "dst",
"type": "address"
},
{
"name": "wad",
"type": "uint256"
}
],
"name": "transfer",
"outputs": [
{
"name": "",
"type": "bool"
}
],
"payable": false,
"stateMutability": "nonpayable",
"type": "function"
},
{
"constant": false,
"inputs": [],
"name": "deposit",
"outputs": [],
"payable": true,
"stateMutability": "payable",
"type": "function"
},
{
"constant": true,
"inputs": [
{
"name": "",
"type": "address"
},
{
"name": "",
"type": "address"
}
],
"name": "allowance",
"outputs": [
{
"name": "",
"type": "uint256"
}
],
"payable": false,
"stateMutability": "view",
"type": "function"
},
{
"payable": true,
"stateMutability": "payable",
"type": "fallback"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"name": "src",
"type": "address"
},
{
"indexed": true,
"name": "guy",
"type": "address"
},
{
"indexed": false,
"name": "wad",
"type": "uint256"
}
],
"name": "Approval",
"type": "event"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"name": "src",
"type": "address"
},
{
"indexed": true,
"name": "dst",
"type": "address"
},
{
"indexed": false,
"name": "wad",
"type": "uint256"
}
],
"name": "Transfer",
"type": "event"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"name": "dst",
"type": "address"
},
{
"indexed": false,
"name": "wad",
"type": "uint256"
}
],
"name": "Deposit",
"type": "event"
},
{
"anonymous": false,
"inputs": [
{
"indexed": true,
"name": "src",
"type": "address"
},
{
"indexed": false,
"name": "wad",
"type": "uint256"
}
],
"name": "Withdrawal",
"type": "event"
}
]
data_updater/find_markets.py
-338
View File
@@ -1,338 +0,0 @@
import pandas as pd
import numpy as np
import os
import requests
import time
import warnings
warnings.filterwarnings("ignore")
if not os.path.exists('data'):
os.makedirs('data')
def get_sel_df(spreadsheet, sheet_name='Selected Markets'):
try:
wk2 = spreadsheet.worksheet(sheet_name)
sel_df = pd.DataFrame(wk2.get_all_records())
sel_df = sel_df[sel_df['question'] != ""].reset_index(drop=True)
return sel_df
except:
return pd.DataFrame()
def get_all_markets(client):
cursor = ""
all_markets = []
while True:
try:
markets = client.get_sampling_markets(next_cursor = cursor)
markets_df = pd.DataFrame(markets['data'])
cursor = markets['next_cursor']
all_markets.append(markets_df)
if cursor is None:
break
except:
break
all_df = pd.concat(all_markets)
all_df = all_df.reset_index(drop=True)
return all_df
def get_bid_ask_range(ret, TICK_SIZE):
bid_from = ret['midpoint'] - ret['max_spread'] / 100
bid_to = ret['best_ask'] #Although bid to this high up will change bid_from because of changing midpoint, take optimistic approach
if bid_to == 0:
bid_to = ret['midpoint']
if bid_to - TICK_SIZE > ret['midpoint']:
bid_to = ret['best_bid'] + (TICK_SIZE + 0.1 * TICK_SIZE)
if bid_from > bid_to:
bid_from = bid_to - (TICK_SIZE + 0.1 * TICK_SIZE)
ask_to = ret['midpoint'] + ret['max_spread'] / 100
ask_from = ret['best_bid']
if ask_from == 0:
ask_from = ret['midpoint']
if ask_from + TICK_SIZE < ret['midpoint']:
ask_from = ret['best_ask'] - (TICK_SIZE + 0.1 * TICK_SIZE)
if ask_from > ask_to:
ask_to = ask_from + (TICK_SIZE + 0.1 * TICK_SIZE)
bid_from = round(bid_from, 3)
bid_to = round(bid_to, 3)
ask_from = round(ask_from, 3)
ask_to = round(ask_to, 3)
if bid_from < 0:
bid_from = 0
if ask_from < 0:
ask_from = 0
return bid_from, bid_to, ask_from, ask_to
def generate_numbers(start, end, TICK_SIZE):
# Calculate the starting point, rounding up to the next hundredth if not an exact multiple of TICK_SIZE
rounded_start = (int(start * 100) + 1) / 100 if start * 100 % 1 != 0 else start + TICK_SIZE
# Calculate the ending point, rounding down to the nearest hundredth
rounded_end = int(end * 100) / 100
# Generate numbers from rounded_start to rounded_end, ensuring they fall strictly within the original bounds
numbers = []
current = rounded_start
while current < end:
numbers.append(current)
current += TICK_SIZE
current = round(current, len(str(TICK_SIZE).split('.')[1])) # Rounding to avoid floating point imprecision
return numbers
def add_formula_params(curr_df, midpoint, v, daily_reward):
curr_df['s'] = (curr_df['price'] - midpoint).abs()
curr_df['S'] = ((v - curr_df['s']) / v) ** 2
curr_df['100'] = 1/curr_df['price'] * 100
curr_df['size'] = curr_df['size'] + curr_df['100']
curr_df['Q'] = curr_df['S'] * curr_df['size']
curr_df['reward_per_100'] = (curr_df['Q'] / curr_df['Q'].sum()) * daily_reward / 2 / curr_df['size'] * curr_df['100']
return curr_df
def process_single_row(row, client):
ret = {}
ret['question'] = row['question']
ret['neg_risk'] = row['neg_risk']
ret['answer1'] = row['tokens'][0]['outcome']
ret['answer2'] = row['tokens'][1]['outcome']
ret['min_size'] = row['rewards']['min_size']
ret['max_spread'] = row['rewards']['max_spread']
token1 = row['tokens'][0]['token_id']
token2 = row['tokens'][1]['token_id']
rate = 0
for rate_info in row['rewards']['rates']:
if rate_info['asset_address'].lower() == '0x2791Bca1f2de4661ED88A30C99A7a9449Aa84174'.lower():
rate = rate_info['rewards_daily_rate']
break
ret['rewards_daily_rate'] = rate
book = client.get_order_book(token1)
bids = pd.DataFrame()
asks = pd.DataFrame()
try:
bids = pd.DataFrame(book.bids).astype(float)
except:
pass
try:
asks = pd.DataFrame(book.asks).astype(float)
except:
pass
try:
ret['best_bid'] = bids.iloc[-1]['price']
except:
ret['best_bid'] = 0
try:
ret['best_ask'] = asks.iloc[-1]['price']
except:
ret['best_ask'] = 0
ret['midpoint'] = (ret['best_bid'] + ret['best_ask']) / 2
TICK_SIZE = row['minimum_tick_size']
ret['tick_size'] = TICK_SIZE
bid_from, bid_to, ask_from, ask_to = get_bid_ask_range(ret, TICK_SIZE)
v = round((ret['max_spread'] / 100), 2)
bids_df = pd.DataFrame()
bids_df['price'] = generate_numbers(bid_from, bid_to, TICK_SIZE)
asks_df = pd.DataFrame()
asks_df['price'] = generate_numbers(ask_from, ask_to, TICK_SIZE)
try:
bids_df = bids_df.merge(bids, on='price', how='left').fillna(0)
except:
bids_df = pd.DataFrame()
try:
asks_df = asks_df.merge(asks, on='price', how='left').fillna(0)
except:
asks_df = pd.DataFrame()
best_bid_reward = 0
ret_bid = pd.DataFrame()
try:
ret_bid = add_formula_params(bids_df, ret['midpoint'], v, rate)
best_bid_reward = round(ret_bid['reward_per_100'].max(), 2)
except:
pass
best_ask_reward = 0
ret_ask = pd.DataFrame()
try:
ret_ask = add_formula_params(asks_df, ret['midpoint'], v, rate)
best_ask_reward = round(ret_ask['reward_per_100'].max(), 2)
except:
pass
ret['bid_reward_per_100'] = best_bid_reward
ret['ask_reward_per_100'] = best_ask_reward
ret['sm_reward_per_100'] = round((best_bid_reward + best_ask_reward) / 2, 2)
ret['gm_reward_per_100'] = round((best_bid_reward * best_ask_reward) ** 0.5, 2)
ret['end_date_iso'] = row['end_date_iso']
ret['market_slug'] = row['market_slug']
ret['token1'] = token1
ret['token2'] = token2
ret['condition_id'] = row['condition_id']
return ret
def get_all_results(all_df, client, max_workers=5):
all_results = []
def process_with_progress(args):
idx, row = args
try:
return process_single_row(row, client)
except:
print("error fetching market")
return None
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = [executor.submit(process_with_progress, (idx, row)) for idx, row in all_df.iterrows()]
for future in concurrent.futures.as_completed(futures):
result = future.result()
if result is not None:
all_results.append(result)
if len(all_results) % (max_workers * 2) == 0:
print(f'{len(all_results)} of {len(all_df)}')
return all_results
def get_combined_markets(new_df, new_markets, sel_df):
if len(sel_df) > 0:
old_markets = new_df[new_df['question'].isin(sel_df['question'])]
all_markets = pd.concat([old_markets, new_markets])
else:
all_markets = new_markets
all_markets = all_markets.drop_duplicates('question')
all_markets = all_markets.sort_values('gm_reward_per_100', ascending=False)
return all_markets
import concurrent.futures
def calculate_annualized_volatility(df, hours):
end_time = df['t'].max()
start_time = end_time - pd.Timedelta(hours=hours)
window_df = df[df['t'] >= start_time]
volatility = window_df['log_return'].std()
annualized_volatility = volatility * np.sqrt(60 * 24 * 252)
return round(annualized_volatility, 2)
def add_volatility(row):
res = requests.get(f'https://clob.polymarket.com/prices-history?interval=1m&market={row["token1"]}&fidelity=10')
price_df = pd.DataFrame(res.json()['history'])
price_df['t'] = pd.to_datetime(price_df['t'], unit='s')
price_df['p'] = price_df['p'].round(2)
price_df.to_csv(f'data/{row["token1"]}.csv', index=False)
price_df['log_return'] = np.log(price_df['p'] / price_df['p'].shift(1))
row_dict = row.copy()
stats = {
'1_hour': calculate_annualized_volatility(price_df, 1),
'3_hour': calculate_annualized_volatility(price_df, 3),
'6_hour': calculate_annualized_volatility(price_df, 6),
'12_hour': calculate_annualized_volatility(price_df, 12),
'24_hour': calculate_annualized_volatility(price_df, 24),
'7_day': calculate_annualized_volatility(price_df, 24 * 7),
'14_day': calculate_annualized_volatility(price_df, 24 * 14),
'30_day': calculate_annualized_volatility(price_df, 24 * 30),
'volatility_price': price_df['p'].iloc[-1]
}
new_dict = {**row_dict, **stats}
return new_dict
def add_volatility_to_df(df, max_workers=2):
results = []
df = df.reset_index(drop=True)
def process_volatility_with_progress(args):
idx, row = args
try:
ret = add_volatility(row.to_dict())
return ret
except:
print("Error fetching volatility")
return None
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = [executor.submit(process_volatility_with_progress, (idx, row)) for idx, row in df.iterrows()]
for future in concurrent.futures.as_completed(futures):
result = future.result()
if result is not None:
results.append(result)
if len(results) % (max_workers * 2) == 0:
print(f'{len(results)} of {len(df)}')
return pd.DataFrame(results)
def get_markets(all_results, sel_df, maker_reward=1):
new_df = pd.DataFrame(all_results)
new_df['spread'] = abs(new_df['best_ask'] - new_df['best_bid'])
new_df = new_df.sort_values('rewards_daily_rate', ascending=False)
new_df[' '] = ''
new_df = new_df[['question', 'answer1', 'answer2', 'neg_risk', 'spread', 'best_bid', 'best_ask', 'rewards_daily_rate', 'bid_reward_per_100', 'ask_reward_per_100', 'gm_reward_per_100', 'sm_reward_per_100', 'min_size', 'max_spread', 'tick_size', 'market_slug', 'token1', 'token2', 'condition_id']]
new_df = new_df.replace([np.inf, -np.inf], 0)
all_data = new_df.copy()
s_df = new_df.copy()
making_markets = s_df[~new_df['question'].isin(sel_df['question'])]
making_markets = making_markets.sort_values('gm_reward_per_100', ascending=False)
making_markets = making_markets[making_markets['gm_reward_per_100'] >= maker_reward]
all_markets = get_combined_markets(new_df, making_markets, sel_df)
return all_data, all_markets
data_updater/google_utils.py
-96
View File
@@ -1,96 +0,0 @@
from google.oauth2.service_account import Credentials
import gspread
import os
import pandas as pd
import requests
import re
def get_spreadsheet(read_only=False):
"""
Get the main Google Spreadsheet using credentials and URL from environment variables
Args:
read_only (bool): If True, uses public CSV export when credentials are missing
"""
spreadsheet_url = os.getenv("SPREADSHEET_URL")
if not spreadsheet_url:
raise ValueError("SPREADSHEET_URL environment variable is not set")
# Check for credentials
if not os.path.exists('credentials.json'):
if read_only:
return ReadOnlySpreadsheet(spreadsheet_url)
else:
raise FileNotFoundError("credentials.json not found. Use read_only=True for read-only access.")
# Normal authenticated access
scope = ["https://spreadsheets.google.com/feeds", "https://www.googleapis.com/auth/drive"]
credentials = Credentials.from_service_account_file('credentials.json', scopes=scope)
client = gspread.authorize(credentials)
spreadsheet = client.open_by_url(spreadsheet_url)
return spreadsheet
class ReadOnlySpreadsheet:
"""Read-only wrapper for Google Sheets using public CSV export"""
def __init__(self, spreadsheet_url):
self.spreadsheet_url = spreadsheet_url
self.sheet_id = self._extract_sheet_id(spreadsheet_url)
def _extract_sheet_id(self, url):
"""Extract sheet ID from Google Sheets URL"""
match = re.search(r'/spreadsheets/d/([a-zA-Z0-9-_]+)', url)
if not match:
raise ValueError("Invalid Google Sheets URL")
return match.group(1)
def worksheet(self, title):
"""Return a read-only worksheet"""
return ReadOnlyWorksheet(self.sheet_id, title)
class ReadOnlyWorksheet:
"""Read-only worksheet that fetches data via CSV export"""
def __init__(self, sheet_id, title):
self.sheet_id = sheet_id
self.title = title
def get_all_records(self):
"""Get all records from the worksheet as a list of dictionaries"""
try:
# Use the public CSV export URL
csv_url = f"https://docs.google.com/spreadsheets/d/{self.sheet_id}/gviz/tq?tqx=out:csv&sheet={self.title}"
response = requests.get(csv_url, timeout=30)
response.raise_for_status()
# Read CSV data into DataFrame
from io import StringIO
df = pd.read_csv(StringIO(response.text))
# Convert to list of dictionaries (same format as gspread)
return df.to_dict('records')
except Exception as e:
print(f"Warning: Could not fetch data from sheet '{self.title}': {e}")
return []
def get_all_values(self):
"""Get all values from the worksheet as a list of lists"""
try:
csv_url = f"https://docs.google.com/spreadsheets/d/{self.sheet_id}/gviz/tq?tqx=out:csv&sheet={self.title}"
response = requests.get(csv_url, timeout=30)
response.raise_for_status()
# Read CSV and return as list of lists
from io import StringIO
df = pd.read_csv(StringIO(response.text))
# Include headers and convert to list of lists
headers = [df.columns.tolist()]
data = df.values.tolist()
return headers + data
except Exception as e:
print(f"Warning: Could not fetch data from sheet '{self.title}': {e}")
return []
data_updater/trading_utils.py
-137
View File
@@ -1,137 +0,0 @@
from py_clob_client.constants import POLYGON
from py_clob_client.client import ClobClient
from py_clob_client.clob_types import OrderArgs, BalanceAllowanceParams, AssetType
from py_clob_client.order_builder.constants import BUY
from web3 import Web3
from web3.middleware import ExtraDataToPOAMiddleware
import json
from dotenv import load_dotenv
load_dotenv()
import time
import os
MAX_INT = 2**256 - 1
def get_clob_client():
host = "https://clob.polymarket.com"
key = os.getenv("PK")
chain_id = POLYGON
if key is None:
print("Environment variable 'PK' cannot be found")
return None
try:
client = ClobClient(host, key=key, chain_id=chain_id)
api_creds = client.create_or_derive_api_creds()
client.set_api_creds(api_creds)
return client
except Exception as ex:
print("Error creating clob client")
print("________________")
print(ex)
return None
def approveContracts():
web3 = Web3(Web3.HTTPProvider("https://polygon-rpc.com"))
web3.middleware_onion.inject(ExtraDataToPOAMiddleware, layer=0)
wallet = web3.eth.account.from_key(os.getenv("PK"))
with open('erc20ABI.json', 'r') as file:
erc20_abi = json.load(file)
ctf_address = "0x4D97DCd97eC945f40cF65F87097ACe5EA0476045"
erc1155_set_approval = """[{"inputs": [{ "internalType": "address", "name": "operator", "type": "address" },{ "internalType": "bool", "name": "approved", "type": "bool" }],"name": "setApprovalForAll","outputs": [],"stateMutability": "nonpayable","type": "function"}]"""
usdc_contract = web3.eth.contract(address="0x2791Bca1f2de4661ED88A30C99A7a9449Aa84174", abi=erc20_abi) # usdc.e
ctf_contract = web3.eth.contract(address=ctf_address, abi=erc1155_set_approval)
for address in ['0x4bFb41d5B3570DeFd03C39a9A4D8dE6Bd8B8982E', '0xC5d563A36AE78145C45a50134d48A1215220f80a', '0xd91E80cF2E7be2e162c6513ceD06f1dD0dA35296']:
usdc_nonce = web3.eth.get_transaction_count( wallet.address )
raw_usdc_txn = usdc_contract.functions.approve(address, int(MAX_INT, 0)).build_transaction({
"chainId": 137,
"from": wallet.address,
"nonce": usdc_nonce
})
signed_usdc_txn = web3.eth.account.sign_transaction(raw_usdc_txn, private_key=os.getenv("PK"))
usdc_tx_receipt = web3.eth.wait_for_transaction_receipt(signed_usdc_txn, 600)
print(f'USDC Transaction for {address} returned {usdc_tx_receipt}')
time.sleep(1)
ctf_nonce = web3.eth.get_transaction_count( wallet.address )
raw_ctf_approval_txn = ctf_contract.functions.setApprovalForAll(address, True).build_transaction({
"chainId": 137,
"from": wallet.address,
"nonce": ctf_nonce
})
signed_ctf_approval_tx = web3.eth.account.sign_transaction(raw_ctf_approval_txn, private_key=os.getenv("PK"))
send_ctf_approval_tx = web3.eth.send_raw_transaction(signed_ctf_approval_tx.raw_transaction)
ctf_approval_tx_receipt = web3.eth.wait_for_transaction_receipt(send_ctf_approval_tx, 600)
print(f'CTF Transaction for {address} returned {ctf_approval_tx_receipt}')
time.sleep(1)
nonce = web3.eth.get_transaction_count( wallet.address )
raw_txn_2 = usdc_contract.functions.approve("0xC5d563A36AE78145C45a50134d48A1215220f80a", int(MAX_INT, 0)).build_transaction({
"chainId": 137,
"from": wallet.address,
"nonce": nonce
})
signed_txn_2 = web3.eth.account.sign_transaction(raw_txn_2, private_key=os.getenv("PK"))
send_txn_2 = web3.eth.send_raw_transaction(signed_txn_2.raw_transaction)
nonce = web3.eth.get_transaction_count( wallet.address )
raw_txn_3 = usdc_contract.functions.approve("0xd91E80cF2E7be2e162c6513ceD06f1dD0dA35296", int(MAX_INT, 0)).build_transaction({
"chainId": 137,
"from": wallet.address,
"nonce": nonce
})
signed_txn_3 = web3.eth.account.sign_transaction(raw_txn_3, private_key=os.getenv("PK"))
send_txn_3 = web3.eth.send_raw_transaction(signed_txn_3.raw_transaction)
def market_action( marketId, action, price, size ):
order_args = OrderArgs(
price=price,
size=size,
side=action,
token_id=marketId,
)
signed_order = get_clob_client().create_order(order_args)
try:
resp = get_clob_client().post_order(signed_order)
print(resp)
except Exception as ex:
print(ex)
pass
def get_position(marketId):
client = get_clob_client()
position_res = client.get_balance_allowance(
BalanceAllowanceParams(
asset_type=AssetType.CONDITIONAL,
token_id=marketId
)
)
orderBook = client.get_order_book(marketId)
price = float(orderBook.bids[-1].price)
shares = int(position_res['balance']) / 1e6
return shares * price
docs/scoping/00-overview.md
+72
View File
@@ -0,0 +1,72 @@
# Poly-Maker v2 — Scoping Overview
**Date: 2026-07-05.** Scope for a ground-up v2 of this Polymarket market-making
bot, targeting political markets.
## Why now (the forcing function)
Polymarket cut over to **CLOB V2 on April 28, 2026** — new order signing, new
exchange contracts, pUSD collateral — and archived the `py-clob-client`
library this repo is built on. **v1 of this bot can no longer place an
order.** We are not refactoring a working system; we're rebuilding a dead one,
which frees us to fix everything at once.
## Goals
1. **CLOB V2 migration** — new client (`py-clob-client-v2`), V2 order signing,
pUSD collateral handling, post-only orders, heartbeat dead-man switch,
batch endpoints. → [03-api-layer.md](03-api-layer.md)
2. **Kill Google Sheets** — config, market selection, and dashboards become
local TOML files + SQLite + a `polymaker` CLI. A laptop with the repo and a
funded wallet is a complete deployment. → [05-config-and-ops.md](05-config-and-ops.md)
3. **Fast market discovery** — Gamma API keyset queries with server-side
politics/liquidity/reward filters replace the hour-long
crawl-every-orderbook scanner: seconds, one process. → [03](03-api-layer.md) §4, [05](05-config-and-ops.md) §2
4. **Much better strategy** — fair-value + inventory-skew quoting, live
volatility/toxicity estimation, regime detection (knowing when *not* to
quote), rewards **and** the new maker-rebates program as explicit objective
terms, and a replay backtester + paper mode to prove it before capital.
→ [04-strategy.md](04-strategy.md)
5. **Maker-only, always** — every order is post-only; exits are repriced
limits, the YES+NO merge loop, and size discipline. No taker orders,
period. → [04](04-strategy.md) §3, §6
6. **Faster & simpler** — one async process (no threads, no Node.js
subprocess, no gc.collect() folklore), uvloop, event-to-order latency
target <50ms internal, Python 3.12+, uv end-to-end, typed and tested.
→ [02-architecture.md](02-architecture.md)
7. **Political markets first, external data later** — v2 trades on
microstructure alone; the architecture reserves an explicit seat (SignalBus)
for polls/news/cross-venue feeds in a later phase.
→ [07-future-external-data.md](07-future-external-data.md)
## Reading order
| Doc | Contents |
|---|---|
| [01-current-state.md](01-current-state.md) | Audit of v1: what exists, what's broken, what dies, what's worth keeping |
| [02-architecture.md](02-architecture.md) | v2 design: components, state machine, package layout, dependencies |
| [03-api-layer.md](03-api-layer.md) | CLOB V2 migration facts, client choice, endpoints, websockets, rate limits, on-chain/merge |
| [04-strategy.md](04-strategy.md) | The quoting engine: FV, vol/toxicity, regimes, rewards+rebates utility, risk, simulation |
| [05-config-and-ops.md](05-config-and-ops.md) | Local config schemas, scanner/CLI, observability, deployment |
| [06-migration-plan.md](06-migration-plan.md) | Phases 0–5, exit criteria, deletion ledger, risk register |
| [07-future-external-data.md](07-future-external-data.md) | Deferred feed layer, scoped so it's additive later |
## Headline decisions (details in the docs)
- **Build in place, delete v1 as we go** — v1 can't trade anyway; no parallel
run. (The v1 code was parked under `legacy/` during the port and has since
been deleted — it lives on in git history on `main`.)
- **Two de-risking spikes before building** (Phase 2): wallet auth on V2 with
our proxy-wallet setup (known SDK bugs around signature types), and live WS
payload capture (schemas changed post-2025 and docs are young).
- **BUY-YES + BUY-NO is the canonical two-sided quote** — USDC(pUSD)-
collateralized on both sides, pairs merge back to cash with locked edge,
and it satisfies the rewards program's two-sidedness rules.
- **Paper-trading gate before live capital**: ≥2 weeks, ≥10 markets, positive
PnL under a conservative fill model, markouts > −0.5 tick at +5m.
## Open questions (tracked in 03 §9)
WS ping/pong contract; `py-sdk` (beta unified SDK) maturity; pUSD
wrap/allowance/merge-collateral mechanics; current `price_change` payload
schema; rebate payout currency; signature-type-2 health in the v2 SDK.
docs/scoping/01-current-state.md
+154
View File
@@ -0,0 +1,154 @@
# 01 — Current State Audit (v1)
Honest inventory of what exists today, what's broken, and what dies in v2.
This is the baseline the rest of the scoping docs build on.
## What v1 is
A single-process Python market maker (~2,700 LOC) plus a separate market-scanner
script, glued together through a **Google Sheet** that acts as config store,
market database, and dashboard at the same time.
```
main.py entry point: REST polling thread + 2 websockets
trading.py the entire strategy (perform_trade, 470 lines)
update_markets.py scanner: crawls all markets, writes to Google Sheets hourly
update_stats.py writes account stats/earnings back to Google Sheets
poly_data/ client wrapper, global mutable state, WS handlers, book
poly_utils/ Google Sheets access (incl. a read-only CSV-export hack)
data_updater/ scanner internals (reward math, volatility fetch)
poly_stats/ earnings/positions summary -> Sheets
poly_merger/ Node.js subprocess to merge YES+NO via Gnosis Safe
positions/*.json ad-hoc risk-off state files written next to the code
```
### Data flow today
1. `update_markets.py` (separate process, ideally separate IP) crawls
`clob.polymarket.com/sampling-markets` page by page, then fetches **one order
book per market** via REST plus one `prices-history` call per market to
compute volatility — thousands of sequential REST calls per scan, ~hourly.
Results are pasted into the "All Markets" / "Volatility Markets" sheets.
2. A human picks rows into the "Selected Markets" sheet and tunes the
"Hyperparameters" sheet.
3. `main.py` re-reads the sheet **every 30 seconds** (network call to Google in
the hot path), polls positions/orders REST every 5s in a background *thread*,
and maintains order books from the market websocket in the asyncio loop.
4. Every `book`/`price_change`/user event spawns `perform_trade(market)` as a
new asyncio task, serialized per market by an `asyncio.Lock`.
## What's structurally wrong
### Concurrency model
- **threading + asyncio mixed**: a daemon thread mutates `global_state` (dicts of
positions/orders) while asyncio tasks read them. No coherent locking
(`global_state.lock` exists but is essentially unused).
- **Blocking I/O inside the event loop**: `py-clob-client` and `requests` are
synchronous. Every order placement/cancel blocks the loop — while it's
signing + POSTing, the bot is blind to every other market's book updates.
- **`perform_trade` holds the per-market lock through a `gc.collect()` and a
hard-coded `await asyncio.sleep(2)`** (trading.py:470). Every quote decision
on a market therefore takes ≥2s and events pile up behind the lock. This
alone caps reactivity at ~0.5 Hz per market in a game where queue position
and reaction to sweeps decide profitability.
- Event storm handling is "spawn a task per WS message" — no coalescing, no
debounce; the lock queue grows unboundedly under load.
### State management
- One module of **global mutable dicts** (`global_state.py`) shared across a
thread and dozens of tasks. Position tracking is a merge of three sources
(data-api REST, user WS fills, on-chain balance) reconciled with the
`performing` set + 15-second staleness hack — a hand-rolled, race-prone
answer to "did my fill land yet?".
- `set_order()` in `poly_data/data_utils.py:136` **overwrites both sides** of a
token's order record with a dict containing only the side that just updated —
the other side's `{price, size}` silently vanishes until the next 5s REST poll.
- Local order book is keyed by condition_id, stores only the YES token book,
derives the NO view by `1 - price`. Fine in principle, but book resync after
WS reconnect relies on the server re-sending a `book` snapshot; there's no
sequence-gap detection or hash check (the WS actually provides a book `hash`).
### Google Sheets as infrastructure
- Config store, market DB, ops dashboard, and inter-process message bus are all
one spreadsheet. It's rate-limited (Sheets API quota), slow (hundreds of ms
per read), needs service-account credentials, and pulls a network dependency
into the trading loop. The "read-only mode" fallback in
`poly_utils/google_utils.py` literally guesses gid numbers 0–4 against a CSV
export URL.
- The scanner and the bot communicate *through* the sheet, so market metadata
in the hot path can be an hour stale (`row['best_bid']` used as a sanity
reference in trading.py:356 comes from the sheet).
### Code quality
- Bare `except:` everywhere (~30 sites), `print()` as the only logging, no
tests at all, no types, dead code left as comments, `gc.collect()` sprinkled
as a superstition (7 call sites), `deets` reused as both the outcome list and
the book snapshot inside the same loop (trading.py:158/197).
- `data_updater/` duplicates `poly_utils` (two `google_utils.py`, two
`trading_utils.py` with different contents).
- Pinned to **Python 3.9** (.python-version = 3.9.18, EOL Oct 2025) and
`py-clob-client==0.28.0`. That client line is **dead**: Polymarket cut over
to CLOB V2 on 2026-04-28 (new order signing, new contracts) and archived
py-clob-client — **v1 of this bot cannot place an order at all anymore**
(see 03). This isn't a refactor of a working bot; it's a rebuild of a
non-functional one.
- A **Node.js subprocess** (`poly_merger/`) exists solely to call
`mergePositions` through a Gnosis Safe — a whole second runtime, package.json
and ethers v5 dependency for one contract call that web3.py can do.
### Strategy weaknesses (why it loses money — detailed fix in 04-strategy.md)
- **Quote logic is penny-jumping**: `bid = best_bid + tick`, `ask = best_ask -
tick`, with a couple of size-based exceptions. No fair-value estimate, no
microprice, no inventory skew — position size only gates *whether* to quote,
never *where*.
- **Adverse selection is handled after the fact**: the only defenses are an
hourly 3-hour volatility number from the spreadsheet and a stop-loss that
**crosses the spread** (sells at best bid — a taker exit, exactly what we
want to ban) then sleeps the market for hours via a JSON file on disk.
- **Rewards are a filter, not an objective**: the scanner computes
reward-per-100 carefully, but the live quoter only checks "is my bid inside
the incentive band" as a boolean. Placement within the band (where the
reward-vs-fill-risk tradeoff actually lives) is never optimized.
- Hard-coded magic everywhere: 0.1–0.9 price band, 250 share absolute cap,
0.005 reprice threshold, 15s staleness, 0.95/0.97/1.01 ratios, `max_size*2`
total-exposure allowance.
- Take-profit reprices only when 2% away from target; sell-side quote sits at
`avgPrice` floor (trading.py / trading_utils.py:136) — i.e. the bot refuses
to exit below cost, which turns losers into resolution lottery tickets.
## What's worth keeping (conceptually)
- **The two-token + merge insight**: buying YES and NO such that both fill and
merging the pair back to USDC is a maker-only exit that never crosses the
spread. v2 promotes this from an afterthought to a core mechanism.
- Book maintenance via `SortedDict` and the YES/NO mirror trick — cheap and
correct.
- The desired-vs-existing order diffing idea in `send_buy_order`/`send_sell_order`
(only cancel/replace on meaningful change) — right instinct, generalized into
a proper reconciler in v2.
- The scanner's reward-per-$100 estimator (`add_formula_params` implements the
official S(v,s) = ((v−s)/v)² scoring) — the math moves into v2's selector.
- The `performing` concept (in-flight fills gate REST reconciliation) — becomes
a real state machine instead of dicts of sets.
## What dies in v2
| Component | Fate |
|---|---|
| Google Sheets (gspread, gspread-dataframe, google-auth, service accounts) | **Deleted.** Local files + SQLite (02/05) |
| `poly_merger/` Node.js + ethers + package.json | **Deleted.** Native Python Safe execution (03) |
| `update_stats.py`, `poly_stats/` (Sheets dashboards) | **Deleted.** Replaced by local status CLI/log (05) |
| `data_updater/` sampling-markets crawl + per-market book fetch | **Deleted.** Gamma API scanner, minutes → seconds (03/05) |
| `global_state.py` module-level dicts | **Deleted.** Typed state store owned by the engine (02) |
| Background REST polling thread | **Deleted.** Single asyncio loop; REST only for reconciliation (02) |
| `positions/*.json` risk files | **Deleted.** SQLite state (02) |
| Python 3.9 pin, py-clob-client 0.28 (archived, V1-only) | Upgraded: 3.12+, py-clob-client-v2 + own async HTTP (03) |
| `print()` logging, bare excepts, gc.collect() | Replaced: structured logging, real error policy (02/05) |
## Repo facts for reference
- uv is **already adopted** (uv.lock, pyproject.toml, hatchling build) — v2
keeps uv and finishes the job: bump `requires-python` to ≥3.12, refresh all
pins, add ruff + mypy + pytest as dev group, `src/` layout.
- Git history is shallow and healthy; no CI exists; MIT licensed; README
already warns the v1 strategy is unprofitable.
docs/scoping/02-architecture.md
+214
View File
@@ -0,0 +1,214 @@
# 02 — v2 Architecture
Single async Python process, local-file config, typed state, no threads, no
Google, no Node. Everything the bot needs to run lives in the repo directory.
## Design principles
1. **One event loop.** All I/O is async. Anything blocking (order signing is
CPU-trivial; EIP-712 signing is ~1ms) stays inline; anything slow-blocking
that can't be made async is banished to `asyncio.to_thread` — but the goal
is zero such call sites.
2. **Websocket-first, REST-reconcile.** The WS feeds are the source of truth
for books, orders, and fills in real time. REST is used for (a) startup
snapshots, (b) periodic drift reconciliation, (c) actions. Never poll REST
for something the WS already pushes.
3. **Desired-state reconciliation.** The strategy never "places an order"; it
emits a *target quote set* per market. The execution layer diffs targets
against live open orders and issues the minimal cancel/place set, applying
tolerances (don't churn for sub-tick / sub-10% size changes) and rate-limit
budgets. This is the v1 `should_cancel` instinct, made a first-class layer.
4. **Local-first ops.** Config = files in the repo. State = SQLite. Dashboards
= CLI. An operator with a laptop and the repo can run, inspect, and tune
everything with no external accounts beyond the Polymarket wallet.
5. **Deterministic and testable.** Strategy is a pure function
`(BookState, Inventory, Params, Clock) -> TargetQuotes`. That makes replay
backtesting and unit tests trivial and is the enabler for 04-strategy.
## Process layout
```
┌────────────────────────────────────────────┐
│ polymaker run │
│ │
Gamma REST ──────▶│ MarketCatalog (refresh ~60s, metadata) │
│ │
CLOB market WS ──▶│ MarketDataService │
│ └─ per-market OrderBook (seq/hash check) │
CLOB user WS ────▶│ UserStream │
│ └─ fills / order lifecycle events │
│ │
│ StateStore (positions, open orders, PnL) │
│ ▲ WS events ▲ REST reconcile │
│ │
│ StrategyEngine │
│ └─ per-market Quoter task (debounced) │
│ emits TargetQuotes │
│ │
│ RiskManager (pre-trade gates, kill switch) │
│ │
│ ExecutionGateway │
│ ├─ order signer (EIP-712, local) │
│ ├─ REST client (async, rate budgeted) │
│ └─ reconciler (target vs live diff) │
│ │
│ Merger (YES+NO -> USDC, web3.py native) │
│ │
│ Persistence: SQLite + JSONL event journal │
│ Telemetry: structlog -> console/file │
└────────────────────────────────────────────┘
```
### Component responsibilities
**MarketCatalog** — market metadata (tokens, tick size, neg-risk flag, reward
params, fees, end date) fetched from Gamma/CLOB REST (see 03). Refreshed on a
slow timer, cached in SQLite. Nothing in the hot path ever awaits this.
**MarketDataService** — owns the market WS connection(s). Maintains one
`OrderBook` per subscribed market: `SortedDict` bids/asks (keep v1's approach),
YES-canonical with NO-side views derived. Handles: snapshot on subscribe,
`price_change` deltas, `tick_size_change`, book-hash verification when
available, and reconnect-with-resnapshot. Exposes sync accessors (microprice,
depth-within-band, imbalance) — pure reads, no awaits — plus an
`asyncio.Event`-style dirty flag per market that wakes the Quoter.
**UserStream** — user WS channel: order placements/cancels/matches and trade
status transitions (MATCHED → MINED → CONFIRMED / FAILED). Feeds StateStore.
Replaces v1's `performing` dicts with an explicit per-order/per-trade state
machine (see below).
**StateStore** — single owner of positions and open orders. In-memory, typed
(dataclasses), persisted to SQLite on change (WAL mode; writes are µs-cheap).
Three inputs, one arbitration rule:
- WS fill events apply immediately (optimistic),
- REST reconciliation (every ~30s, and on demand after anomalies) corrects
drift *only* for tokens with no in-flight trades,
- on-chain balance is consulted only by the Merger before merging.
This is v1's `performing`/`last_trade_update`/`avgOnly` logic, but as one
explicit component with tests instead of three dicts and a timeout.
**StrategyEngine / Quoter** — one lightweight task per market. Loop:
```
wait for (book dirty | fill | timer tick | param change), coalesced
debounce: recompute at most once per N ms (default 200ms), immediately on fill
compute TargetQuotes = strategy(book, inventory, params, clock) # pure
hand to RiskManager -> ExecutionGateway
```
No lock held during I/O; the Quoter never awaits network calls itself.
**RiskManager** — pre-trade checks (per-market notional cap, global exposure
cap, neg-risk group cap, max open order value, price sanity band vs recent
mid), plus global circuit breakers (drawdown kill switch, WS staleness halt,
API error-rate halt). Owns the "reduce-only" and "halt" market modes. Detailed
policy in 04-strategy.md §Risk.
**ExecutionGateway** — the only component that talks to the CLOB REST API for
actions. Async HTTP (httpx), local EIP-712 signing (V2 order format), a
token-bucket rate budgeter tuned to documented API limits (03), batch
place/cancel endpoints, **post-only on every quote**, and the target-vs-live
reconciler with churn tolerances. Also owns the **exchange heartbeat** (03):
a ~5s keepalive; if the process dies or partitions, the exchange cancels all
our orders within ~10s — the dead-man switch a maker-only bot must have.
Every request/response is journaled.
**Merger** — pure-Python replacement for poly_merger (03 §Merge). Triggered by
StateStore when min(YES,NO) ≥ merge threshold; runs as a low-priority task,
never blocks quoting.
**Persistence** — `state.db` (SQLite): market catalog cache, positions, order
history, fills, PnL marks, risk events. `journal/*.jsonl`: append-only raw
event log (WS messages in, orders out) — this is also the capture format the
backtester replays (04 §Simulation).
**Telemetry** — structlog JSON to file + human console; `polymaker status`
CLI reads SQLite for positions/PnL/quotes (05). Optional later: Prometheus
exporter. No Sheets dashboards.
## Order/trade state machine
Replaces `performing` + timestamps + 15s cleanup:
```
Order: DRAFT -> SIGNED -> POSTED -> LIVE -> {PARTIALLY_FILLED} -> DONE
└─ REJECTED └─ CANCELED
Trade: MATCHED -> MINED -> CONFIRMED
└────────────-> FAILED (position rolled back, reconcile forced)
```
Timeouts are per-transition (e.g. MATCHED without MINED after T → force REST
reconcile for that token) instead of one global 15s sweep.
## Package layout
```
poly-maker/
├── pyproject.toml # uv-managed; python >=3.12
├── uv.lock
├── config/
│ ├── config.toml # bot settings: wallet env refs, global risk, engine knobs
│ ├── strategy.toml # named parameter profiles (replaces Hyperparameters sheet)
│ └── markets.toml # selected markets + per-market overrides (replaces Selected Markets)
├── src/polymaker/
│ ├── cli.py # `polymaker` entrypoint: run/scan/markets/status/flatten/...
│ ├── config.py # pydantic-settings models, TOML load, hot-reload (watchfiles)
│ ├── catalog/ # Gamma/CLOB market discovery + scoring (05)
│ ├── marketdata/ # WS client, order book, book analytics
│ ├── userstream/ # user WS, event parsing
│ ├── state/ # StateStore, models, SQLite persistence
│ ├── strategy/ # pure quoting logic + parameter models (04)
│ ├── risk/ # RiskManager, circuit breakers
│ ├── execution/ # signer, REST client, rate budgeter, reconciler
│ ├── merge.py # native merge (03)
│ ├── journal.py # JSONL event journal
│ └── sim/ # replay backtester + paper-trading mode (04)
├── tests/
└── docs/scoping/
```
`poly_data`, `poly_utils`, `poly_stats`, `data_updater`, `poly_merger`,
`trading.py`, `main.py`, `update_markets.py`, `update_stats.py` are all
deleted once their replacements land (06 has the phasing).
## Concurrency & performance notes
- Python 3.12+ (faster asyncio, per-interpreter GIL groundwork irrelevant here
— one loop is enough; this workload is I/O bound with tiny CPU bursts).
- `uvloop` as the event loop (free ~2-4x loop throughput on macOS/Linux).
- WS message → book update → quoter wake is all sync code on the loop; the
only awaits in the hot path are the WS `recv` and the outbound HTTP.
- Signing: `py_order_utils`/eth-account key ops are ~1ms — sign inline, no
executor.
- Target: event-to-order-POST latency < 50ms internally (network to Polymarket
will dominate at ~50–150ms; colocating the box geographically near the CLOB
(AWS us-east) is an ops choice, noted in 05).
- Memory: no gc.collect() calls; books for ~50 markets are a few MB.
## Error policy
- No bare excepts. Each component defines recoverable (log + retry with
backoff) vs fatal (halt quoting, keep cancel capability, alert) errors.
- On any uncertainty about our own open-order/position state → freeze new
quotes for that market, force reconcile, resume. "When confused, cancel and
resync" is the universal recovery.
- On process crash: the exchange heartbeat has already cancelled our orders
within ~10s; startup sequence is cancel-all (belt) → REST snapshot → resume.
## Dependency set (target)
| Purpose | Package |
|---|---|
| HTTP | httpx (async) |
| WS | websockets (current) |
| Signing/order building | **py-clob-client-v2** (V1 client is archived/dead); direct py_order_utils + eth-account if we outgrow it (03) |
| Chain (merge, balances) | web3.py 7.x |
| Config/validation | pydantic v2 + pydantic-settings, tomllib |
| Hot reload | watchfiles |
| State | sqlite3 stdlib (+ tiny DAO layer; no ORM) |
| Logging | structlog |
| CLI | typer + rich |
| Loop | uvloop |
| Dev | ruff, mypy, pytest, pytest-asyncio |
pandas/numpy leave the runtime hot path (they remain fine inside the scanner
and sim analysis). gspread, gspread-dataframe, google-auth, py-clob-client
(v1), and the entire Node/ethers runtime go; sortedcontainers stays (book).
docs/scoping/03-api-layer.md
+203
View File
@@ -0,0 +1,203 @@
# 03 — API Layer: V1 → V2 Migration
Polymarket executed a **hard, non-backward-compatible cutover to CLOB V2 on
April 28, 2026**. V1-signed orders stopped working that day; the legacy
`py-clob-client` this repo pins (0.28.0; latest 0.34.6) was **archived May 11,
2026** and cannot trade against V2. There is no incremental path — the API
layer is a rewrite, which is exactly what this scoping assumes.
Primary sources: [V2 migration guide](https://docs.polymarket.com/v2-migration),
[changelog](https://docs.polymarket.com/changelog),
[llms.txt doc index](https://docs.polymarket.com/llms.txt).
## 1. What changed in CLOB V2
| Area | V1 (this repo) | V2 (target) |
|---|---|---|
| Base URL | `https://clob.polymarket.com` | same URL, new backend |
| Order struct | `nonce`, `feeRateBps`, `taker` fields | removed; adds `timestamp` (ms), `metadata`, `builder`; fees set by protocol at match time |
| EIP-712 domain | Exchange version "1" | version **"2"**, new verifying contracts |
| Exchange contracts | `0x4bFb41d5...` / neg-risk `0xC5d563A3...` | CTF Exchange V2 `0xE111180000d2663C0091e4f400237545B87B996B`; Neg Risk CTF Exchange V2 `0xe2222d279d744050d28e00520010520000310F59` |
| Collateral | USDC.e (`0x2791Bca1...`) | **pUSD** (ERC-20, 1:1 USDC-backed); API traders wrap via the Collateral Onramp contract and set pUSD allowances |
| Auth | L1 EIP-712 + L2 HMAC headers | **unchanged**; existing API keys carried over |
| Fees | fields existed, mostly zero | **makers pay zero**; taker-only fees `fee = C · feeRate · p·(1−p)` per category (Politics 1.00%, geopolitics/world-events free); live rate from each market's `feeSchedule` — never hardcode |
| Neg-risk | PartialCreateOrderOptions flag | fully supported; neg-risk orders sign against the neg-risk V2 contract (SDK handles) |
### Consequences for us
- **Wallet/allowance setup is a new ops step**: wrap USDC→pUSD via the onramp,
approve pUSD to the V2 exchange (buys) and conditional tokens (sells).
Balance checks read pUSD, not USDC.e (v1's hardcoded USDC contract in
`polymarket_client.py:78` dies).
- **Signature-type risk**: v1 runs `signature_type=2` (Polymarket proxy/Safe
funder). `py-clob-client-v2` has a cluster of open auth bugs around
signature types 2/3 and Safe wallets (issues #70/#85/#87/#90/#91) and
"invalid order version" migration confusion (#92). **Phase 2 starts with a
wallet-auth spike**: place/cancel one order with our exact wallet setup
before building anything on top. Fallback: EOA wallet (signature_type=0).
- Fine ticks now exist (0.001, even 0.0025); `tick_size_change` events (fires
crossing 0.96/0.04) must be handled live or orders start rejecting.
## 2. Client library choice
| Option | Status | Call |
|---|---|---|
| `py-clob-client` 0.34.x | archived, V1-only | **dead — remove** |
| **`py-clob-client-v2`** (1.0.2, Jul 2026) | official successor, active | **use for Phase 2.** Sync-only → wrap calls we need in a thin async adapter, or use it only for signing/building and POST via httpx ourselves |
| `polymarket-client` (py-sdk) | official unified SDK, **beta**, sync+async | watch; adopt when stable if its async client is solid — re-evaluate at Phase 2 start |
| Direct: `py_order_utils` + eth-account + httpx | max control | fallback if the v2 client's sync-ness or Safe bugs block us; order signing is a well-documented EIP-712 struct |
Decision: **start with `py-clob-client-v2` for order building/signing, own the
HTTP layer with httpx** (async, connection-pooled, our rate budgeter, our
retries). The signing code is pure CPU and composes fine with our loop. This
also keeps us one small step from going fully direct if the SDK disappoints.
## 3. Endpoints v2 uses (and the v1 habits they replace)
### Trading (CLOB, L2 auth)
- `POST /order` — single post-only/GTC/GTD order.
- **`POST /orders` (batch, ≤15)** — quote both tokens × layers in one request;
v1 placed orders one blocking call at a time.
- `DELETE /orders` (≤1,000 ids) — targeted mass cancel; replaces v1's
cancel-all-per-asset churn.
- `DELETE /cancel-market-orders`, `DELETE /cancel-all` — recovery paths
(startup, kill switch).
- **`POST /heartbeats`** (Jan 2026): opt-in dead-man switch — miss a ~10s
heartbeat window and the exchange cancels all our orders. **Mandatory for
v2**: a maker-only book of stale quotes after a crash/network partition is
the biggest tail risk we have. ExecutionGateway sends one every ~5s.
- **Order types**: GTC, GTD, FOK, FAK, and **Post-Only (Jan 2026, GTC/GTD
only)** — the enforcement mechanism for our maker-only mandate (04 §3.2);
every quote goes out post-only.
### Market data (REST, no auth)
- `GET /book`, **`POST /books` (batch)**, `/midpoint(s)`, `/price(s)` —
snapshots at startup/resync. The scanner uses batch `/books` instead of v1's
one-book-per-market crawl.
- `GET /prices-history` — still exists; only used by the scanner for coarse
historical vol (live vol comes from our own WS-derived estimates).
- `GET /sampling-markets` — replaced by Gamma for discovery (below); CLOB
market metadata (`min_incentive_size`, `max_incentive_spread`, tick size,
neg_risk) still fetched per selected market as authoritative.
### Rewards & earnings
- Liquidity-rewards program is live and mechanically the same family as v1:
score `S = ((v−s)/v)²`, adjusted midpoint (dust-filtered), two-sided
`Q_min = max(min(Q_one,Q_two), max(Q_one/c, Q_two/c))` with c=3.0,
per-minute sampling, daily payout, $1/market minimum, **double-sided
quoting required to score at all when mid < 0.10 or > 0.90**.
[docs](https://docs.polymarket.com/developers/market-makers/liquidity-rewards)
- **Maker Rebates Program (new, Jan 2026)**: ~20–25% of taker fees collected
in fee-enabled categories (politics 25%-ish) rebated daily to that market's
makers pro-rata. A second income stream v1 never knew about — enters the
strategy utility function (04 §4) and the market selector's scoring (05).
- Earnings actuals: rewards user endpoints (v1's `poly_stats` hit a
polymarket.com internal API with L2 headers; v2 uses the documented rewards
endpoints) — pulled daily into SQLite for calibration.
### Relayer (only if we keep Safe-style wallets)
- Relayer `POST /submit` now returns `{transactionID, state}` — poll
`GET /transaction` for the hash. Affects the merge flow if executed via
proxy wallet (§6).
## 4. Market discovery: Gamma API (the "better, faster market list")
v1's scanner crawled `sampling-markets` page by page then fetched **one order
book + one price history per market** — thousands of sequential calls, ~hourly
cadence, results pasted into Google Sheets. v2 replaces the whole pipeline
with Gamma (`https://gamma-api.polymarket.com`, no auth):
- **`GET /markets/keyset`** (Apr 2026, cursor-paginated, limit ≤100; the
offset endpoints are slated for deprecation — build keyset-first) with
server-side filters: `active`, `closed` (defaults false now), `tag_id` (+
`related_tags`) for **politics**, `liquidity_num_min`, `volume_num_min`,
`end_date_min/max`, `rewards_min_size`.
- One response row already carries what v1 burned two extra calls per market
to compute: `bestBid`, `bestAsk`, `spread`, `liquidityNum`, `volumeNum` (+
24h/1wk/1mo), `rewardsMinSize`, `rewardsMaxSpread`, `feeSchedule`/`feesEnabled`,
`orderPriceMinTickSize`, `orderMinSize`, `negRisk`, `clobTokenIds`,
`conditionId`, `endDate`.
- Tag IDs are numeric — resolve once via `GET /tags`, cache in SQLite.
- A politics-filtered scan is **a handful of paginated requests (seconds)**
vs. v1's ~hour; run it every few minutes if we like. Rate limits are ample
(Gamma 4,000 req/10s global, `/markets` 300/10s).
- Depth/reward-optimality checks for the shortlist only: batch `POST /books`
on candidate tokens.
- Event grouping for neg-risk risk caps comes from `GET /events/keyset`
(markets nest their event; events give us the sibling-outcome structure).
## 5. WebSockets
Same hosts as v1 (`wss://ws-subscriptions-clob.polymarket.com/ws/{market,user}`)
— **URLs unchanged by the V2 cutover** — but the payloads moved:
- **Market channel**: subscribe with
`{"assets_ids": [...], "type": "market", "custom_feature_enabled": true}`.
Events: `book` (snapshot), `price_change` (**schema changed Sept 2025 —
now includes best_bid/best_ask; re-derive the parser from current docs**),
`tick_size_change`, `last_trade_price` (includes `fee_rate_bps`), and with
the custom flag: **`best_bid_ask`, `new_market`, `market_resolved`**.
`market_resolved` is a free early-resolution alarm for the risk ladder
(04 §6). The old ~100-token subscription cap was removed (May 2025).
- **User channel**: auth payload as v1, but scope subscriptions with
`"markets": [condition_ids]`. `order` + `trade` events, trade status ladder
`MATCHED → MINED → CONFIRMED / RETRYING / FAILED` (maps directly onto the
02 state machine).
- **Ping/pong**: reportedly server pings every 5s with a 10s pong deadline —
not confirmed on the official channel pages; **verify empirically** and make
keepalive parameters config, not constants.
- **RTDS** (`wss://ws-live-data.polymarket.com`, Sept 2025): separate
real-time data service — `activity` (all trades/matches platform-wide),
`comments`, crypto/equity prices, `clob_user` topic. Not needed for the
core loop; the `activity` topic is a cheap future enhancement for
cross-market flow signals (still Polymarket-internal data, so it doesn't
violate the no-external-feeds rule; noted in 07).
## 6. On-chain layer (merge + balances) — Node.js dies
- **Merger rewrite in Python** (web3.py 7.x, which the repo already ships):
`mergePositions` on the ConditionalTokens contract (plain markets) or the
NegRiskAdapter (neg-risk), exactly what `poly_merger/merge.js` does through
ethers v5. For proxy/Safe wallets, execute through the Safe with
`safe-eth-py`, or via Polymarket's relayer client — decide in the Phase 2
wallet spike (same spike as §1 signature-type risk; if we land on an EOA,
the Safe code path disappears entirely).
- **pUSD questions to resolve in the spike** (docs are young): pUSD token
address; whether V2-era markets' CTF collateral is pUSD (merge returns pUSD)
and whether pre-V2 political markets still merge to USDC.e; onramp
wrap/unwrap mechanics and gas. The Merger and the balance sheet code treat
collateral as config-driven per market, not a constant.
- Balances: pUSD ERC-20 for cash, CTF `balanceOf` for positions (unchanged),
data-api `/positions` for cross-checks (rate limit 150/10s — reconcile use
only).
## 7. Rate budget (per official limits, Jun 2026)
Relevant ceilings: `POST /order` 5,000/10s burst; batch `POST /orders`
2,000/10s; `DELETE /orders` 2,000/10s; `cancel-market-orders` 1,500/10s;
books/prices 500–1,500/10s; Cloudflare throttles (queues) rather than 429s —
so the real risk is **silent latency injection**, not errors.
For ~50 markets × 2 tokens × 2 layers ≈ 200 live orders, a full re-quote is
~14 batch calls — nothing. The ExecutionGateway still gets a token-bucket
budgeter (config: fraction of documented limits, default 25%) because
throttle-queueing would silently add latency exactly when the market is
moving; the budgeter surfaces pressure as a metric + shed-load policy
(skip reprices whose edge < threshold first) instead.
## 8. Out of scope
- **Polymarket US** (docs.polymarket.us): separate CFTC-regulated venue,
separate API/auth/SDKs. Not targeted; noted for the future in 07.
- Builder attribution (`builderCode`): not needed unless we monetize order
flow; field exists in the order struct if ever wanted.
## 9. Open items to verify at implementation time
1. WS ping/pong contract (empirical test).
2. `py-sdk` maturity re-check at Phase 2 start (async client + WS coverage).
3. pUSD address / onramp mechanics / merge collateral (wallet spike).
4. Exact current `price_change` + `best_bid_ask` payload schemas.
5. Maker-rebate payout currency (USDC vs pUSD) — accounting only.
6. Whether heartbeats interact with batch cancels/cancel-all on reconnect.
7. Signature type 2 (proxy/Safe) health in `py-clob-client-v2` (issue tracker
before the spike).
docs/scoping/04-strategy.md
+292
View File
@@ -0,0 +1,292 @@
# 04 — Strategy v2: Maker-Only Quoting for Political Markets
The v1 strategy is penny-jumping with an after-the-fact stop loss. v2 replaces
it with a quoting engine built around four ideas:
1. a **fair value estimate** derived purely from market microstructure (no
external data in this phase — see 07 for the future feed layer),
2. **inventory-skewed, volatility-scaled quotes** that never cross the spread,
3. **liquidity rewards as an explicit term in the objective**, not a boolean
filter,
4. **regime detection** that decides when *not* to quote — which is where most
of the money is saved.
Strategy code is a pure function `(BookState, Inventory, Params, Clock) ->
TargetQuotes`, unit-testable and replayable (see §9).
## 0. The market we're fitting
Political markets on Polymarket have a specific microstructure profile:
- **Binary, bounded [0,1]**, usually inside **neg-risk event groups** (multi-
candidate events where Σ YES = 1 by construction).
- **Long-dated with punctuated equilibrium**: weeks of low-vol mean-reversion
interrupted by violent news repricing (debates, indictments, dropouts,
election nights). The flow between events is heavily retail/directional
(favorable for a maker); the flow *during* events is fast and toxic.
- **Early-resolution risk**: a candidate dropping out gates a market to 0
instantly. There is no external feed to warn us in v2 — microstructure
(sweeps, book evaporation) is the alarm, and size caps are the survival
mechanism.
- Depth is deep at the touch on flagship markets, thin on down-ballot ones —
parameters must scale with observed book depth, not fixed constants.
Everything below is designed against this profile.
## 1. Fair value (FV) engine
All inputs are local: the book we maintain from the WS and the trade tape.
- **Microprice**: depth-weighted mid over the top K levels (K≈3), i.e. mid
pulled toward the thin side. Robust to size-1 dust orders via per-level
minimum-size filtering (keep v1's `find_best_price_with_size` idea, tuned by
observed depth instead of hardcoded 100/20 shares).
- **Flow adjustment**: EWMA of signed aggressor volume (from `last_trade_price`
/trade events) over ~1–5 min nudges FV in the direction of persistent flow.
Political markets trend on news; leaning with flow reduces adverse fills.
- **Cross-outcome consistency (neg-risk)**: Σ of YES microprices across the
event should ≈ 1. Persistent deviation → books dislocated (news repricing in
progress) → contributes to regime score (§5); large deviation is also the
trigger for the optional arb module (§7).
- **Clamping**: FV is smoothed (short EWMA) and clamped against jumps unless
the jump is confirmed by trades, not just quote flicker.
Explicitly *not* an input in v2: polls, news, other venues (see 07).
## 2. Volatility & toxicity engine
Replaces the spreadsheet's hour-stale "3_hour volatility" with live estimates:
- **Realized vol**: EWMA of squared FV returns at three horizons (~10s, ~1m,
~15m), computed on the WS stream. σ_short drives spread width; the
short/long ratio drives regime detection.
- **Sweep detector**: single aggression that consumes ≥N levels or moves FV by
≥X ticks → immediate event flag (§5).
- **Markout tracker** (self-tuning core): for every one of our fills, record
FV at +30s and +5m. Per-market EWMA of markout = realized adverse-selection
cost. Consistently negative markouts automatically widen that market's
spread / reduce its size via a toxicity multiplier. This is how the bot
learns "this market eats me" without any human or external data.
## 3. Quote construction (maker-only)
### 3.1 Reservation price and half-spread
```
skew = γ · σ_short · (q / q_max) # in price units
r = FV − skew # reservation price
δ = max(δ_min, c_vol·σ_short + c_tox·toxicity)
bid_raw = r − δ ; ask_raw = r + δ
```
- `q` = net inventory in the YES token (NO inventory counts negative),
`q_max` = per-market cap. Long inventory pushes both quotes down: buy
cheaper, sell more eagerly. This single mechanism replaces v1's pile of
size-gating special cases.
- Fees: on CLOB V2 **makers pay zero** — no maker fee term in δ. Taker fees
(politics 1.00%, per-market `feeSchedule` from the catalog, 03) matter to
us twice: they fund the **maker rebates** income term (§4), and they enter
the cost model of the optional set-arb module (§7) only.
- All prices then snap to tick and go through **placement logic**:
### 3.2 Placement: join, don't jump
- Default: place at the better of (raw price, one tick behind the touch) —
i.e. **join the level or sit behind it**; jump the queue (improve the touch)
only when `FV − best_ask` / `best_bid − FV` edge exceeds tick + costs. v1
penny-jumps unconditionally, which wins queue position precisely when the
price is about to be wrong.
- **Never cross**: candidate bid ≥ best_ask ⇒ reprice to best_ask − tick (and
symmetric for asks), validated against the live book at send time. Enforced
at the exchange too: **every quote is a Post-Only order** (added to CLOB
Jan 2026, GTC/GTD — 03), which rejects rather than takes. Maker-only is a
property of the system, not a convention.
- **Min edge vs FV**: bid ≤ FV − δ_edge_min always; we never pay through fair
value for queue position or rewards.
- **Layering**: optionally split size across 2–3 levels per side (touch-join +
deeper levels). Sweeps fill the deep levels at prices that already include
the move; layered in-band size also raises the reward score (§4).
- **Churn control**: reprice only if |target − live| > reprice_ticks or size
drift > 15%; batch cancel+place; respect the rate budget (03). Queue
position is an asset — don't burn it for sub-tick prettiness.
### 3.3 Two-token quoting and the merge loop
On Polymarket, BUY NO @ q is economically SELL YES @ 1−q, but it is
collateralized by USDC rather than by YES inventory. v2 makes the **BUY-YES +
BUY-NO pair the canonical two-sided quote**:
- Both quotes are bids (USDC-collateralized) → we can quote two-sided with
zero inventory, from day one, on both sides of the reward band.
- When both fill: hold YES @ p and NO @ q with p + q < 1 → **merge** pair back
to 1 USDC. Realized edge = 1 − p − q, captured without ever selling. The
merge (native Python, 03) runs opportunistically in the background.
- When inventory accumulates on one leg, exits are dual-routed, still maker:
(a) keep/raise the opposite-token bid (merge exit), or
(b) place a SELL limit on the held token above FV.
The quoter maintains whichever route has the better expected time-to-fill ×
price; often both, sized to not double-exit.
### 3.4 Sizing
- Base size per market from config; scaled down by toxicity, by inventory
utilization (q/q_max), and near end-date (§6).
- Respect `orderMinSize`; low-price multiplier logic from v1 (more shares when
price < 0.10) is replaced by sizing in **USDC notional**, constant per quote
regardless of price level.
- No hardcoded 0.1–0.9 price band: quote anywhere the reward band and risk
checks allow, but size shrinks near the boundaries where payoff asymmetry is
extreme (long-shot bias zone; near-certain markets pay ~nothing to make).
## 4. Rewards and rebates as an objective
Two live income programs (03 §3), both maker-side, both enter the objective:
- **Liquidity rewards** (same family v1 targeted, confirmed live 2026):
per-market daily rate, sampled ~per minute; score `S = ((v − s)/v)²` per
order (s = distance from the dust-filtered *adjusted midpoint*, v =
max_spread), min-size gate, two-sided rule
`Q_min = max(min(Q_one, Q_two), max(Q_one/c, Q_two/c))` with c = 3.0, and —
important — **when mid < 0.10 or > 0.90 only double-sided liquidity scores
at all**. The formula already lives in v1's scanner (`add_formula_params`);
v2 moves it into the strategy as a live utility term.
- **Maker rebates** (new Jan 2026): ~25% of taker fees collected in politics
markets is redistributed daily to that market's makers pro-rata. Unlike
rewards (paid for *resting* in-band), rebates are paid for *being filled* —
they directly subsidize adverse selection and shift the optimizer toward
tighter quoting in fee-enabled markets with heavy taker flow.
```
utility(placement) = E[spread capture] + E[reward $/day at this placement]
+ E[rebate | fill] − E[adverse selection | regime]
− inventory penalty
```
Consequences the optimizer discovers on its own:
- **Quiet regime** → sit deeper inside the band with more size (reward-dense,
low fill probability, low AS cost) — "farming posture".
- **Hot regime** → rewards can't compensate expected AS loss → widen beyond
the band or pull entirely. v1 could never do this; its reward check was a
one-way "must be inside band" constraint that *forced* it to quote tight
exactly when it shouldn't.
- **Two-sidedness**: the Qmin rule (and the hard double-sided requirement at
extreme mids) pays for balanced quoting — another reason the
BUY-YES/BUY-NO pair posture (§3.3) is the default; it keeps both sides
in-band even while carrying inventory.
- Income accounting: actual reward + rebate payouts pulled daily (03) and
attributed per market, so the utility model's estimates are calibrated
against reality instead of drifting.
## 5. Regime detection (when not to quote)
Per-market state machine, evaluated on every quoter wake:
| Regime | Trigger (any) | Behavior |
|---|---|---|
| QUIET | default | farming posture: in-band, layered, full size |
| TRENDING | flow EWMA persistent one-sided; σ_short/σ_long elevated | lean FV with flow, widen δ, half size |
| EVENT | sweep detector; FV jump > J ticks; neg-risk Σ dislocation; book depth evaporates | **pull all quotes**, cooloff T_c (~30–120s), re-enter at 2–3× δ decaying back to normal |
| REDUCE_ONLY | inventory ≥ hard cap; end-date proximity; risk manager order | exit quotes only |
| HALTED | WS stale > T; market flagged closed/`acceptingOrders=false`; kill switch | cancel all, no quoting |
The EVENT → re-enter-wide → decay pattern replaces v1's "stop-loss then sleep
4 hours via JSON file". Cooldowns are seconds-to-minutes and price-aware, not
hours and blind.
## 6. Inventory, exits, and lifecycle risk — without taker orders
Maker-only exit means **we cannot rely on being able to get out**. The
controls compensate:
- **Soft cap `q_soft`**: skew (§3.1) already leans quotes; beyond q_soft the
adding side is pulled entirely, exit side gains urgency: exit price decays
from FV + δ toward one tick above best bid over τ_urgency (still maker,
never crossing). Urgency accelerates if FV drifts against the position.
- **Hard cap `q_max`**: reduce-only. Sizing (§3.4) makes approaching q_max
progressively harder, so the hard stop is rarely the active constraint.
- **No cost-basis anchoring**: v1 refused to ask below avgPrice, converting
losses into resolution lotteries. v2 prices exits off FV and urgency only;
cost basis is reporting, not strategy.
- **Merge always-on**: min(YES, NO) ≥ threshold → merge to USDC (frees
capital, realizes locked edge, zero market impact).
- **End-date ladder** (from catalog metadata): T−D days → size taper begins;
T−H hours → REDUCE_ONLY; T−S → HALTED + cancel. Political markets also
resolve early: the market WS now pushes **`market_resolved` / `new_market`
events** (03 §5) — instant halt on resolution — with catalog refresh
(closed/`acceptingOrders` flags) and the EVENT regime as backstops.
- **Per-market and global risk** (RiskManager, 02): market notional cap,
neg-risk **event-group worst-case-loss cap** (multiple candidates in one
event are one bet, not N), total exposure cap, daily realized-loss kill
switch (halts quoting, keeps cancel/exit capability), stale-data halt.
## 7. Neg-risk structural opportunities (optional module, off by default)
Political events are neg-risk groups; the structure occasionally gifts money:
- **Set arbitrage**: if Σ best-asks of all outcomes < 1 − ε, buy the set
(maker where possible, but this is the one place taker math is ever
justified — behind a config flag, default off, consistent with the
maker-only policy).
- **NegRiskAdapter conversions**: NO(i) sets convert to YES(others) + USDC —
enables inventory rebalancing across outcomes without trading. Scoped in 03
(§contracts); implementation phase 4+.
- Both are opportunistic add-ons; core PnL is quoting + rewards. Scoped now so
the state model (event-group awareness) is built in from day one.
## 8. Parameters
`config/strategy.toml` holds named profiles (replaces the Hyperparameters
sheet); `markets.toml` maps markets → profile + overrides (05 has schemas).
Illustrative:
```toml
[profiles.political-longdated]
gamma = 0.5 # inventory risk aversion (skew strength)
delta_min_ticks = 2
c_vol = 1.2 # spread per unit short-vol
c_tox = 2.0 # spread per unit toxicity score
q_max_usdc = 500 # hard inventory cap, notional
q_soft_frac = 0.6
layers = 2
reprice_ticks = 2
debounce_ms = 200
event_cooloff_s = 60
end_date_taper_d = 7
reduce_only_h = 24
```
Every v1 magic number (0.005 reprice, 250 cap, 0.95/0.97/1.01, 15s, 0.1–0.9
band, `max_size*2`) either becomes a named documented parameter or is deleted
by design. Hot-reload via watchfiles; changes apply on next quoter wake, no
restart.
## 9. Simulation, paper trading, and measurement
The strategy is only "much better" if we can prove it. Three rungs:
1. **Replay backtest** (`polymaker sim replay`): the journal (02) captures raw
WS streams; the sim reconstructs books and runs the strategy with a queue-
position fill model — conservative mode fills our order only when the tape
prints *through* our price; optimistic mode fills pro-rata at the touch.
Truth lies between; both are reported. Reward income is computed from the
scoring formula on our simulated quotes. Start capturing journals from the
moment Phase 2 lands, so weeks of political-market data exist before the
strategy ships.
2. **Paper mode** (`polymaker run --paper`): full live pipeline, orders
journaled but not posted; fills simulated from the live tape. The gate for
live capital.
3. **Live KPIs** (05 §status): per-market markout curves (+30s/+5m), fill
ratio, two-sided in-band uptime (the reward KPI), reward $ vs estimate,
realized/unrealized PnL, inventory duration, EVENT-regime frequency.
Acceptance criteria to go live (Phase 4 gate, see 06): paper-mode ≥ 2 weeks on
≥ 10 political markets with (a) positive net PnL including simulated rewards
under the **conservative** fill model, (b) average +5m markout > −0.5 tick,
(c) zero risk-limit breaches.
## 10. Explicit non-goals in v2 (deferred to 07)
- External data: polls, news, social monitors, other venues (Kalshi et al).
- Taker execution of any kind (except the flagged neg-risk set-arb, default off).
- Cross-event statistical arbitrage.
- Sports/crypto-specific logic — the engine is generic, but tuning and the
selector target political markets first.
docs/scoping/05-config-and-ops.md
+132
View File
@@ -0,0 +1,132 @@
# 05 — Local Config, Market Selection, and Operations
Google Sheets currently plays four roles: config store, market database,
selection UI, and dashboard. v2 replaces each with a local equivalent. The
test: **a laptop with the repo, a `.env`, and a funded wallet is a complete
deployment.** No Google account, no service-account JSON, no second IP running
a scanner.
## 1. Configuration files (replaces Hyperparameters + Selected Markets sheets)
All TOML, all in `config/`, all validated by pydantic at load, all
hot-reloadable via watchfiles (applied at the next quoter wake; invalid edits
are rejected with a logged diff, keeping the last good config — an edit can
never crash the bot).
### `config/config.toml` — engine & account
```toml
[wallet]
# secrets stay in .env: PK, BROWSER_ADDRESS; this file only references them
signature_type = 2 # revisit after Phase 2 wallet spike (03 §1)
[engine]
debounce_ms = 200
reconcile_interval_s = 30
heartbeat = true # exchange dead-man switch (03 §3)
journal = true # raw WS/order journal for the backtester
[risk] # global — per-market caps live in markets.toml
max_total_exposure_usdc = 5000
max_event_group_loss_usdc = 1000 # neg-risk group worst-case
daily_loss_kill_usdc = 250
ws_stale_halt_s = 10
```
### `config/strategy.toml` — named parameter profiles
Schema in 04 §8. Profiles (`political-longdated`, `political-hot`, …) replace
the Hyperparameters sheet's `param_type` groups.
### `config/markets.toml` — the trade list (replaces Selected Markets sheet)
```toml
[[markets]]
slug = "will-x-win-the-2028-democratic-nomination"
profile = "political-longdated"
q_max_usdc = 800 # optional per-market overrides of the profile
[[markets]]
condition_id = "0x37a6de..." # slug or condition_id both accepted
profile = "political-hot"
enabled = false # keep the entry, stop quoting
```
The bot resolves slugs → tokens/metadata via the catalog at startup and on
reload. Adding a market while running = edit file, save; the engine
subscribes, seeds the book, starts quoting. Removing/disabling = cancel that
market's orders, unsubscribe. This is the entire "market management UI".
`.env` keeps only secrets: `PK`, `BROWSER_ADDRESS`, optional RPC URL override.
`SPREADSHEET_URL` and `credentials.json` are deleted.
## 2. Market catalog & scanner (replaces data_updater + All/Volatility Markets sheets)
`state.db` (SQLite) holds the market catalog; the scanner is a subcommand, not
a second deployment:
- `polymaker scan` — Gamma keyset sweep with server-side filters (03 §4):
politics tag(s), active, min liquidity/volume, end-date window, rewards
present. Enriches the shortlist with batch `/books` depth and reward-density
estimates (the v1 `add_formula_params` math, kept) **plus the maker-rebate
estimate** (taker-fee volume × rebate share — 03 §3). Writes to SQLite.
Seconds per run; scheduled inside the bot process (e.g. every 15 min) —
the "run on a different IP" advice dies with the crawl that motivated it.
- `polymaker markets` — rank/browse the catalog in the terminal (rich table):
`polymaker markets --tag politics --min-reward 20 --max-vol-sum 20 --sort score`.
Score = expected daily income (rewards + rebates) vs. volatility & spread
risk — v1's `gm_reward_per_100` / `volatility_sum` composite, recalibrated.
- `polymaker markets add <slug> [--profile P]` — appends to `markets.toml`
(file remains the source of truth; the command is a convenience editor).
- Volatility for ranking: `prices-history` coarse estimate at scan time;
markets we quote get live WS-derived vol which supersedes it.
## 3. Runtime CLI (replaces update_stats.py + Summary sheet + eyeballing prints)
Single `polymaker` entrypoint (typer):
| Command | Purpose |
|---|---|
| `polymaker run [--paper]` | start the engine (paper mode: full pipeline, no order POSTs) |
| `polymaker status` | positions, open quotes, inventory, PnL, regime per market (reads SQLite; works while the bot runs) |
| `polymaker pnl [--daily]` | realized/unrealized PnL, reward + rebate income vs. estimates |
| `polymaker flatten [market]` | reduce-only mode (maker exits) for one/all markets |
| `polymaker cancel-all` | panic button (also runs automatically at startup) |
| `polymaker scan` / `markets` | catalog (above) |
| `polymaker sim replay ...` | backtester (04 §9) |
| `polymaker doctor` | preflight: wallet auth spike checks, pUSD balance/allowances, WS reachability, clock skew |
## 4. Observability
- **structlog** JSON logs to `logs/` + human-readable console. Every order
decision logs its inputs (FV, σ, inventory, regime) — a quote is always
explainable after the fact.
- **Journal** (`journal/*.jsonl`): raw WS in / orders out. Feeds the
backtester and post-mortems.
- **SQLite** is the queryable operational record: fills, orders, PnL marks,
reward payouts, risk events. Any dashboard we want later (Grafana, a small
web page) reads this — no scoping dependency on it now.
- **Alerting (minimal, phase 4)**: kill-switch trips, WS-stale halts,
reconcile divergence, heartbeat failures → log at CRITICAL + optional
webhook URL in config (generic; Discord/Telegram/ntfy all take a POST).
- KPI set defined in 04 §9 (markouts, two-sided in-band uptime, fill ratio,
income vs. estimate).
## 5. Deployment & runtime ops
- **uv end-to-end** (already adopted; finish the job): `requires-python =
">=3.12"`, refreshed lock, `uv run polymaker ...`, dev group with ruff,
mypy, pytest, pytest-asyncio. Delete the 3.9 pin.
- **Placement**: any always-on Linux box; latency to the CLOB (AWS us-east
region) is the one infra knob that matters for queue position — a small VPS
in us-east-1 beats a laptop on wifi. Not required for correctness.
- **Supervision**: systemd unit example in the README (`Restart=always`);
startup sequence is cancel-all → snapshot → resume, so a restart is always
safe. The exchange heartbeat covers the gap between death and restart.
- **CI (GitHub Actions)**: ruff + mypy + pytest on PR. No deploy pipeline
needed for a single-operator bot.
- **State backup**: `state.db` + `config/` are the whole world; journal grows
~MBs/day per market — rotate + optionally compress with age.
## 6. Security notes
- `PK` only ever in `.env` (gitignored) / environment; never in TOML, logs,
or SQLite. Log wallet addresses truncated.
- The CLI never prints secrets; `doctor` verifies without echoing.
- Dependencies pinned via uv.lock; renovate/dependabot optional later.
docs/scoping/06-migration-plan.md
+109
View File
@@ -0,0 +1,109 @@
# 06 — Migration Plan
Phased so that every phase leaves the repo in a working, testable state, and
the risky unknowns (wallet auth on V2, pUSD) are attacked first. v1 cannot
trade anyway (CLOB V1 is gone — 03), so there is no parallel-run constraint:
we are building v2 in place, deleting v1 as replacements land.
## Phase 0 — Repo hygiene & toolchain (small)
- `src/polymaker/` package skeleton (02 layout), typer CLI stub.
- pyproject: `requires-python = ">=3.12"`, `.python-version` → 3.12.x, fresh
`uv lock`; dev group: ruff, mypy, pytest, pytest-asyncio; ruff replaces
black; GitHub Actions running lint+types+tests.
- Delete immediately (nothing depends on them working): `poly_stats/`,
`update_stats.py`, the read-only Sheets CSV hack, dead commented code.
The rest of the v1 code (`trading.py`/`main.py`/`data_updater/` …) was parked
under `legacy/` during the port and has now been deleted — it survives only in
git history on `main`.
- Remove deps: gspread, gspread-dataframe, google-auth. Add: httpx, pydantic,
structlog, typer, rich, watchfiles, uvloop, py-clob-client-v2.
**Exit:** `uv run polymaker --help` works; CI green; no Google imports left.
## Phase 1 — Catalog, config, scanner (replaces Sheets as data source)
- pydantic config models + TOML loading + hot reload (05 §1).
- Gamma keyset client, tag resolution, SQLite catalog, market scoring
(rewards + rebates + vol), `polymaker scan` / `markets` / `markets add`
(05 §2).
- CLOB public data client (books batch, midpoints, prices-history) for
enrichment.
**Exit:** politics markets scanned, ranked, and selected into `markets.toml`
in seconds, with no Google anywhere. (v1 bot is now un-runnable — fine, see
above.)
## Phase 2 — Execution core (the V2 unknowns live here — do the spikes first)
- **Spike A (days, throwaway code): wallet auth on V2.** Wrap USDC→pUSD,
set allowances, place + cancel one post-only order with our exact wallet
(signature_type 2), read it back on the user WS. Resolves 03 §9 items 3/7.
Decision output: keep proxy wallet vs. move to EOA; py-clob-client-v2 vs.
direct signing.
- **Spike B: WS contract.** Capture live `book`/`price_change`/`best_bid_ask`
/`tick_size_change` payloads (custom features on), confirm ping/pong,
write the parsers against reality.
- Then build: MarketDataService (books + analytics + resync), UserStream,
StateStore + order/trade state machine, ExecutionGateway (signer, httpx,
rate budgeter, heartbeats, batch ops, reconciler), Merger (native Python,
Safe-or-EOA per Spike A), journal.
- Integration test: on 1–2 quiet live markets with $10-size manual targets,
prove place/cancel/fill/merge/reconcile round-trips, then heartbeat-driven
auto-cancel on kill.
**Exit:** engine can hold a hand-specified quote set live, survive WS drops,
crashes, and restarts safely. Journal capture running 24/7 from here on (the
backtester's data supply).
## Phase 3 — Strategy engine (04) + simulation
- Pure strategy module: FV, vol/toxicity, regime machine, quote construction,
reward/rebate utility, inventory/exit logic — with unit tests per component.
- Replay backtester over the journals accumulated since Phase 2; parameter
study for the political profiles.
- Paper mode (`run --paper`) end-to-end.
- Delete the parked v1 code — the port is done. (Done: `legacy/` removed.)
**Exit:** paper trading ≥2 weeks, ≥10 political markets, meeting the
acceptance gates from 04 §9 (positive PnL under conservative fills, markouts
> −0.5 tick at +5m, zero risk breaches).
## Phase 4 — Live rollout + ops hardening
- RiskManager fully wired (caps, kill switches, event-group limits), alert
webhook, `status`/`pnl`/`flatten`/`doctor` polished.
- Go live: 2–3 markets at minimum size → widen coverage/size on weekly review
of the KPI set. us-east VPS placement.
- Reward/rebate reconciliation against actual daily payouts.
**Exit:** stable unattended operation; income attribution (spread vs rewards
vs rebates) reported per market.
## Phase 5 — Deferred (separately scoped)
- External data feeds (07): polls, news, cross-venue.
- Neg-risk set-arb + NegRiskAdapter conversions (04 §7) if the structural
edge shows up in captured data.
- Optional: RTDS `activity` flow features, Prometheus/Grafana, Polymarket US
venue.
## Deletion ledger (what's gone by the end of Phase 3)
`trading.py`, `main.py`, `update_markets.py`, `update_stats.py`, `poly_data/`,
`poly_utils/`, `poly_stats/`, `data_updater/`, `poly_merger/` (entire Node
runtime), `positions/` runtime dir, `credentials.json` expectations,
`SPREADSHEET_URL`, Python 3.9 pin, gspread/gspread-dataframe/google-auth,
py-clob-client (v1), pandas-in-hot-path, all `print()` logging, all bare
excepts, all `gc.collect()` calls.
## Risk register
| Risk | Mitigation |
|---|---|
| V2 SDK auth bugs with proxy/Safe wallets | Spike A first; EOA fallback decided in days, not after building |
| pUSD mechanics undocumented corners | Spike A covers wrap/allowance/merge collateral; treat collateral as per-market config |
| WS payload drift (young V2 docs) | Spike B parses from captured reality; parsers tolerate unknown fields |
| Maker-only exits too slow in a news jump | Position caps sized for survivable worst case (04 §6); EVENT regime pulls quotes early; accept this as the strategy's core tradeoff |
| Rewards program parameters shift | All reward math data-driven from per-market API fields; utility recalibrated from actual payouts |
| Strategy underperforms | Paper-gate before capital; journal-driven iteration loop is the product as much as the bot |
docs/scoping/07-future-external-data.md
+63
View File
@@ -0,0 +1,63 @@
# 07 — Future: External Data Feeds (deferred, scoped now)
v2 trades political markets on **microstructure alone** (04 §1). That is a
deliberate constraint, not a belief that external data has no value — the
architecture reserves a seat for it so adding feeds later is additive, not a
refactor. This doc scopes the seat and the candidate feeds; **none of this is
built in v2.**
## Architectural seat (built in v2, empty)
The strategy signature already is `(BookState, Inventory, Params, Clock) ->
TargetQuotes`. The extension is one optional input:
```
SignalBus: name -> {value, confidence, as_of, ttl}
```
- Strategy reads signals as **FV adjusters and regime inputs** with explicit
staleness: a signal past its TTL contributes nothing (fail-safe to pure
microstructure — the bot must never be worse than v2 because a feed died).
- Feed adapters are separate async tasks (or even separate processes writing
to the bus via SQLite/socket) — never in the quote path; the hot loop only
reads a dict.
- Every signal is journaled like WS data, so the replay backtester (04 §9)
can grade any feed's marginal value before it touches live quotes. **A feed
earns its way in through replay evidence, or it doesn't ship.**
## Candidate feeds for political markets (rough priority)
1. **Event calendar (highest value / lowest effort)**: debates, primaries,
election nights, court dates, scheduled rulings. Even a hand-maintained
`calendar.toml` (date, market tags, severity) lets the regime machine
pre-widen/pull *before* scheduled volatility instead of reacting to the
first sweep. No API needed to start.
2. **Cross-venue prices**: Kalshi (public API), and Polymarket US as it
grows. Same-event price divergence is both an FV prior and a toxicity
warning (fast flow often arbs venues; if Kalshi moved and we didn't, our
quote is stale). Clean REST/WS, well-bounded work.
3. **Poll aggregates / forecast models**: RCP/538-style averages, Silver
Bulletin, model outputs. Slow-moving FV anchor for long-dated markets —
mostly useful as a *sanity band* (flag when market FV drifts far from
model FV) rather than a quoting signal.
4. **News/headline triggers**: wire headlines, X/Twitter firehose, Google
Trends spikes. Highest alpha, highest noise and engineering cost;
realistically an "instant EVENT-regime trigger" (pull quotes on keyword
burst), not a pricing input. Last in line.
5. **Polymarket-internal extras** (not really "external", could land earlier):
RTDS `activity` topic for platform-wide flow/toxicity features, comment
velocity per market as an attention proxy (03 §5).
## Explicitly out of scope until then
- Any taker execution driven by signals (feeds inform *maker* quoting only —
the maker-only policy survives the feed era).
- LLM/news-summarization pipelines, sentiment models.
- Trading the Polymarket US venue (separate API/regulatory surface — 03 §8).
## Preconditions before starting this phase
- v2 live and stable through at least one high-volatility political event.
- Journal + replay infrastructure proven (it is the evaluation harness here).
- KPI baseline established, so each feed's marginal contribution is measured
against a known control.
main.py
-115
View File
@@ -1,115 +0,0 @@
import gc # Garbage collection
import time # Time functions
import asyncio # Asynchronous I/O
import traceback # Exception handling
import threading # Thread management
from poly_data.polymarket_client import PolymarketClient
from poly_data.data_utils import update_markets, update_positions, update_orders
from poly_data.websocket_handlers import connect_market_websocket, connect_user_websocket
import poly_data.global_state as global_state
from poly_data.data_processing import remove_from_performing
from dotenv import load_dotenv
load_dotenv()
def update_once():
"""
Initialize the application state by fetching market data, positions, and orders.
"""
update_markets() # Get market information from Google Sheets
update_positions() # Get current positions from Polymarket
update_orders() # Get current orders from Polymarket
def remove_from_pending():
"""
Clean up stale trades that have been pending for too long (>15 seconds).
This prevents the system from getting stuck on trades that may have failed.
"""
try:
current_time = time.time()
# Iterate through all performing trades
for col in list(global_state.performing.keys()):
for trade_id in list(global_state.performing[col]):
try:
# If trade has been pending for more than 15 seconds, remove it
if current_time - global_state.performing_timestamps[col].get(trade_id, current_time) > 15:
print(f"Removing stale entry {trade_id} from {col} after 15 seconds")
remove_from_performing(col, trade_id)
print("After removing: ", global_state.performing, global_state.performing_timestamps)
except:
print("Error in remove_from_pending")
print(traceback.format_exc())
except:
print("Error in remove_from_pending")
print(traceback.format_exc())
def update_periodically():
"""
Background thread function that periodically updates market data, positions and orders.
- Positions and orders are updated every 5 seconds
- Market data is updated every 30 seconds (every 6 cycles)
- Stale pending trades are removed each cycle
"""
i = 1
while True:
time.sleep(5) # Update every 5 seconds
try:
# Clean up stale trades
remove_from_pending()
# Update positions and orders every cycle
update_positions(avgOnly=True) # Only update average price, not position size
update_orders()
# Update market data every 6th cycle (30 seconds)
if i % 6 == 0:
update_markets()
i = 1
gc.collect() # Force garbage collection to free memory
i += 1
except:
print("Error in update_periodically")
print(traceback.format_exc())
async def main():
"""
Main application entry point. Initializes client, data, and manages websocket connections.
"""
# Initialize client
global_state.client = PolymarketClient()
# Initialize state and fetch initial data
global_state.all_tokens = []
update_once()
print("After initial updates: ", global_state.orders, global_state.positions)
print("\n")
print(f'There are {len(global_state.df)} market, {len(global_state.positions)} positions and {len(global_state.orders)} orders. Starting positions: {global_state.positions}')
# Start background update thread
update_thread = threading.Thread(target=update_periodically, daemon=True)
update_thread.start()
# Main loop - maintain websocket connections
while True:
try:
# Connect to market and user websockets simultaneously
await asyncio.gather(
connect_market_websocket(global_state.all_tokens),
connect_user_websocket()
)
print("Reconnecting to the websocket")
except:
print("Error in main loop")
print(traceback.format_exc())
await asyncio.sleep(1)
gc.collect() # Clean up memory
if __name__ == "__main__":
asyncio.run(main())
poly_data/CONSTANTS.py
-3
View File
@@ -1,3 +0,0 @@
# Minimum position size to trigger position merging
# Positions smaller than this will be ignored to save on gas costs
MIN_MERGE_SIZE = 20
poly_data/abis.py
-3
View File
File diff suppressed because one or more lines are too long
poly_data/data_processing.py
-160
View File
@@ -1,160 +0,0 @@
import json
from sortedcontainers import SortedDict
import poly_data.global_state as global_state
import poly_data.CONSTANTS as CONSTANTS
from trading import perform_trade
import time
import asyncio
from poly_data.data_utils import set_position, set_order, update_positions
def process_book_data(asset, json_data):
global_state.all_data[asset] = {
'asset_id': json_data['asset_id'], # token_id for the Yes token
'bids': SortedDict(),
'asks': SortedDict()
}
global_state.all_data[asset]['bids'].update({float(entry['price']): float(entry['size']) for entry in json_data['bids']})
global_state.all_data[asset]['asks'].update({float(entry['price']): float(entry['size']) for entry in json_data['asks']})
def process_price_change(asset, side, price_level, new_size, asset_id=None):
# Skip updates for the No token to prevent duplicated updates
# Only process if asset_id matches the stored asset_id for this market
if asset_id and asset in global_state.all_data and asset_id != global_state.all_data[asset]['asset_id']:
return
if side == 'bids':
book = global_state.all_data[asset]['bids']
else:
book = global_state.all_data[asset]['asks']
if new_size == 0:
if price_level in book:
del book[price_level]
else:
book[price_level] = new_size
def process_data(json_datas, trade=True):
# Ensure input is always a list
if isinstance(json_datas, dict):
json_datas = [json_datas]
for json_data in json_datas:
event_type = json_data['event_type']
asset = json_data['market']
if event_type == 'book':
process_book_data(asset, json_data)
if trade:
asyncio.create_task(perform_trade(asset))
elif event_type == 'price_change':
for data in json_data['price_changes']:
side = 'bids' if data['side'] == 'BUY' else 'asks'
price_level = float(data['price'])
new_size = float(data['size'])
# Pass asset_id if available in the data
asset_id = data.get('asset_id', None)
process_price_change(asset, side, price_level, new_size, asset_id)
if trade:
asyncio.create_task(perform_trade(asset))
# pretty_print(f'Received book update for {asset}:', global_state.all_data[asset])
def add_to_performing(col, id):
if col not in global_state.performing:
global_state.performing[col] = set()
if col not in global_state.performing_timestamps:
global_state.performing_timestamps[col] = {}
# Add the trade ID and track its timestamp
global_state.performing[col].add(id)
global_state.performing_timestamps[col][id] = time.time()
def remove_from_performing(col, id):
if col in global_state.performing:
global_state.performing[col].discard(id)
if col in global_state.performing_timestamps:
global_state.performing_timestamps[col].pop(id, None)
def process_user_data(rows):
for row in rows:
market = row['market']
side = row['side'].lower()
token = row['asset_id']
if token in global_state.REVERSE_TOKENS:
col = token + "_" + side
if row['event_type'] == 'trade':
size = 0
price = 0
maker_outcome = ""
taker_outcome = row['outcome']
is_user_maker = False
for maker_order in row['maker_orders']:
if maker_order['maker_address'].lower() == global_state.client.browser_wallet.lower():
print("User is maker")
size = float(maker_order['matched_amount'])
price = float(maker_order['price'])
is_user_maker = True
maker_outcome = maker_order['outcome'] #this is curious
if maker_outcome == taker_outcome:
side = 'buy' if side == 'sell' else 'sell' #need to reverse as we reverse token too
else:
token = global_state.REVERSE_TOKENS[token]
if not is_user_maker:
size = float(row['size'])
price = float(row['price'])
print("User is taker")
print("TRADE EVENT FOR: ", row['market'], "ID: ", row['id'], "STATUS: ", row['status'], " SIDE: ", row['side'], " MAKER OUTCOME: ", maker_outcome, " TAKER OUTCOME: ", taker_outcome, " PROCESSED SIDE: ", side, " SIZE: ", size)
if row['status'] == 'CONFIRMED' or row['status'] == 'FAILED' :
if row['status'] == 'FAILED':
print(f"Trade failed for {token}, decreasing")
asyncio.create_task(asyncio.sleep(2))
update_positions()
else:
remove_from_performing(col, row['id'])
print("Confirmed. Performing is ", len(global_state.performing[col]))
print("Last trade update is ", global_state.last_trade_update)
print("Performing is ", global_state.performing)
print("Performing timestamps is ", global_state.performing_timestamps)
asyncio.create_task(perform_trade(market))
elif row['status'] == 'MATCHED':
add_to_performing(col, row['id'])
print("Matched. Performing is ", len(global_state.performing[col]))
set_position(token, side, size, price)
print("Position after matching is ", global_state.positions[str(token)])
print("Last trade update is ", global_state.last_trade_update)
print("Performing is ", global_state.performing)
print("Performing timestamps is ", global_state.performing_timestamps)
asyncio.create_task(perform_trade(market))
elif row['status'] == 'MINED':
remove_from_performing(col, row['id'])
elif row['event_type'] == 'order':
print("ORDER EVENT FOR: ", row['market'], " STATUS: ", row['status'], " TYPE: ", row['type'], " SIDE: ", side, " ORIGINAL SIZE: ", row['original_size'], " SIZE MATCHED: ", row['size_matched'])
set_order(token, side, float(row['original_size']) - float(row['size_matched']), row['price'])
asyncio.create_task(perform_trade(market))
else:
print(f"User date received for {market} but its not in")
poly_data/data_utils.py
-176
View File
@@ -1,176 +0,0 @@
import poly_data.global_state as global_state
from poly_data.utils import get_sheet_df
import time
import poly_data.global_state as global_state
#sth here seems to be removing the position
def update_positions(avgOnly=False):
pos_df = global_state.client.get_all_positions()
for idx, row in pos_df.iterrows():
asset = str(row['asset'])
if asset in global_state.positions:
position = global_state.positions[asset].copy()
else:
position = {'size': 0, 'avgPrice': 0}
position['avgPrice'] = row['avgPrice']
if not avgOnly:
position['size'] = row['size']
else:
for col in [f"{asset}_sell", f"{asset}_buy"]:
#need to review this
if col not in global_state.performing or not isinstance(global_state.performing[col], set) or len(global_state.performing[col]) == 0:
try:
old_size = position['size']
except:
old_size = 0
if asset in global_state.last_trade_update:
if time.time() - global_state.last_trade_update[asset] < 5:
print(f"Skipping update for {asset} because last trade update was less than 5 seconds ago")
continue
if old_size != row['size']:
print(f"No trades are pending. Updating position from {old_size} to {row['size']} and avgPrice to {row['avgPrice']} using API")
position['size'] = row['size']
else:
print(f"ALERT: Skipping update for {asset} because there are trades pending for {col} looking like {global_state.performing[col]}")
global_state.positions[asset] = position
def get_position(token):
token = str(token)
if token in global_state.positions:
return global_state.positions[token]
else:
return {'size': 0, 'avgPrice': 0}
def set_position(token, side, size, price, source='websocket'):
token = str(token)
size = float(size)
price = float(price)
global_state.last_trade_update[token] = time.time()
if side.lower() == 'sell':
size *= -1
if token in global_state.positions:
prev_price = global_state.positions[token]['avgPrice']
prev_size = global_state.positions[token]['size']
if size > 0:
if prev_size == 0:
# Starting a new position
avgPrice_new = price
else:
# Buying more; update average price
avgPrice_new = (prev_price * prev_size + price * size) / (prev_size + size)
elif size < 0:
# Selling; average price remains the same
avgPrice_new = prev_price
else:
# No change in position
avgPrice_new = prev_price
global_state.positions[token]['size'] += size
global_state.positions[token]['avgPrice'] = avgPrice_new
else:
global_state.positions[token] = {'size': size, 'avgPrice': price}
print(f"Updated position from {source}, set to ", global_state.positions[token])
def update_orders():
all_orders = global_state.client.get_all_orders()
orders = {}
if len(all_orders) > 0:
for token in all_orders['asset_id'].unique():
if token not in orders:
orders[str(token)] = {'buy': {'price': 0, 'size': 0}, 'sell': {'price': 0, 'size': 0}}
curr_orders = all_orders[all_orders['asset_id'] == str(token)]
if len(curr_orders) > 0:
sel_orders = {}
sel_orders['buy'] = curr_orders[curr_orders['side'] == 'BUY']
sel_orders['sell'] = curr_orders[curr_orders['side'] == 'SELL']
for type in ['buy', 'sell']:
curr = sel_orders[type]
if len(curr) > 1:
print("Multiple orders found, cancelling")
global_state.client.cancel_all_asset(token)
orders[str(token)] = {'buy': {'price': 0, 'size': 0}, 'sell': {'price': 0, 'size': 0}}
elif len(curr) == 1:
orders[str(token)][type]['price'] = float(curr.iloc[0]['price'])
orders[str(token)][type]['size'] = float(curr.iloc[0]['original_size'] - curr.iloc[0]['size_matched'])
global_state.orders = orders
def get_order(token):
token = str(token)
if token in global_state.orders:
if 'buy' not in global_state.orders[token]:
global_state.orders[token]['buy'] = {'price': 0, 'size': 0}
if 'sell' not in global_state.orders[token]:
global_state.orders[token]['sell'] = {'price': 0, 'size': 0}
return global_state.orders[token]
else:
return {'buy': {'price': 0, 'size': 0}, 'sell': {'price': 0, 'size': 0}}
def set_order(token, side, size, price):
curr = {}
curr = {side: {'price': 0, 'size': 0}}
curr[side]['size'] = float(size)
curr[side]['price'] = float(price)
global_state.orders[str(token)] = curr
print("Updated order, set to ", curr)
def update_markets():
received_df, received_params = get_sheet_df()
if len(received_df) > 0:
# Ensure multiplier column exists and fill NaN values with empty string
if 'multiplier' not in received_df.columns:
received_df['multiplier'] = ''
else:
received_df['multiplier'] = received_df['multiplier'].fillna('')
global_state.df, global_state.params = received_df.copy(), received_params
for _, row in global_state.df.iterrows():
for col in ['token1', 'token2']:
row[col] = str(row[col])
if row['token1'] not in global_state.all_tokens:
global_state.all_tokens.append(row['token1'])
if row['token1'] not in global_state.REVERSE_TOKENS:
global_state.REVERSE_TOKENS[row['token1']] = row['token2']
if row['token2'] not in global_state.REVERSE_TOKENS:
global_state.REVERSE_TOKENS[row['token2']] = row['token1']
for col2 in [f"{row['token1']}_buy", f"{row['token1']}_sell", f"{row['token2']}_buy", f"{row['token2']}_sell"]:
if col2 not in global_state.performing:
global_state.performing[col2] = set()
poly_data/global_state.py
-49
View File
@@ -1,49 +0,0 @@
import threading
import pandas as pd
# ============ Market Data ============
# List of all tokens being tracked
all_tokens = []
# Mapping between tokens in the same market (YES->NO, NO->YES)
REVERSE_TOKENS = {}
# Order book data for all markets
all_data = {}
# Market configuration data from Google Sheets
df = None
# ============ Client & Parameters ============
# Polymarket client instance
client = None
# Trading parameters from Google Sheets
params = {}
# Lock for thread-safe trading operations
lock = threading.Lock()
# ============ Trading State ============
# Tracks trades that have been matched but not yet mined
# Format: {"token_side": {trade_id1, trade_id2, ...}}
performing = {}
# Timestamps for when trades were added to performing
# Used to clear stale trades
performing_timestamps = {}
# Timestamps for when positions were last updated
last_trade_update = {}
# Current open orders for each token
# Format: {token_id: {'buy': {price, size}, 'sell': {price, size}}}
orders = {}
# Current positions for each token
# Format: {token_id: {'size': float, 'avgPrice': float}}
positions = {}
poly_data/polymarket_client.py
-320
View File
@@ -1,320 +0,0 @@
from dotenv import load_dotenv # Environment variable management
import os # Operating system interface
# Polymarket API client libraries
from py_clob_client.client import ClobClient
from py_clob_client.clob_types import OrderArgs, BalanceAllowanceParams, AssetType, PartialCreateOrderOptions
from py_clob_client.constants import POLYGON
# Web3 libraries for blockchain interaction
from web3 import Web3
from web3.middleware import ExtraDataToPOAMiddleware
from eth_account import Account
import requests # HTTP requests
import pandas as pd # Data analysis
import json # JSON processing
import subprocess # For calling external processes
from py_clob_client.clob_types import OpenOrderParams
# Smart contract ABIs
from poly_data.abis import NegRiskAdapterABI, ConditionalTokenABI, erc20_abi
# Load environment variables
load_dotenv()
class PolymarketClient:
"""
Client for interacting with Polymarket's API and smart contracts.
This class provides methods for:
- Creating and managing orders
- Querying order book data
- Checking balances and positions
- Merging positions
The client connects to both the Polymarket API and the Polygon blockchain.
"""
def __init__(self, pk='default') -> None:
"""
Initialize the Polymarket client with API and blockchain connections.
Args:
pk (str, optional): Private key identifier, defaults to 'default'
"""
host="https://clob.polymarket.com"
# Get credentials from environment variables
key=os.getenv("PK")
browser_address = os.getenv("BROWSER_ADDRESS")
# Don't print sensitive wallet information
print("Initializing Polymarket client...")
chain_id=POLYGON
self.browser_wallet=Web3.to_checksum_address(browser_address)
# Initialize the Polymarket API client
self.client = ClobClient(
host=host,
key=key,
chain_id=chain_id,
funder=self.browser_wallet,
signature_type=2
)
# Set up API credentials
self.creds = self.client.create_or_derive_api_creds()
self.client.set_api_creds(creds=self.creds)
# Initialize Web3 connection to Polygon
web3 = Web3(Web3.HTTPProvider("https://polygon-rpc.com"))
web3.middleware_onion.inject(ExtraDataToPOAMiddleware, layer=0)
# Set up USDC contract for balance checks
self.usdc_contract = web3.eth.contract(
address="0x2791Bca1f2de4661ED88A30C99A7a9449Aa84174",
abi=erc20_abi
)
# Store key contract addresses
self.addresses = {
'neg_risk_adapter': '0xd91E80cF2E7be2e162c6513ceD06f1dD0dA35296',
'collateral': '0x2791Bca1f2de4661ED88A30C99A7a9449Aa84174',
'conditional_tokens': '0x4D97DCd97eC945f40cF65F87097ACe5EA0476045'
}
# Initialize contract interfaces
self.neg_risk_adapter = web3.eth.contract(
address=self.addresses['neg_risk_adapter'],
abi=NegRiskAdapterABI
)
self.conditional_tokens = web3.eth.contract(
address=self.addresses['conditional_tokens'],
abi=ConditionalTokenABI
)
self.web3 = web3
def create_order(self, marketId, action, price, size, neg_risk=False):
"""
Create and submit a new order to the Polymarket order book.
Args:
marketId (str): ID of the market token to trade
action (str): "BUY" or "SELL"
price (float): Order price (0-1 range for prediction markets)
size (float): Order size in USDC
neg_risk (bool, optional): Whether this is a negative risk market. Defaults to False.
Returns:
dict: Response from the API containing order details, or empty dict on error
"""
# Create order parameters
order_args = OrderArgs(
token_id=str(marketId),
price=price,
size=size,
side=action
)
signed_order = None
# Handle regular vs negative risk markets differently
if neg_risk == False:
signed_order = self.client.create_order(order_args)
else:
signed_order = self.client.create_order(order_args, options=PartialCreateOrderOptions(neg_risk=True))
try:
# Submit the signed order to the API
resp = self.client.post_order(signed_order)
return resp
except Exception as ex:
print(ex)
return {}
def get_order_book(self, market):
"""
Get the current order book for a specific market.
Args:
market (str): Market ID to query
Returns:
tuple: (bids_df, asks_df) - DataFrames containing bid and ask orders
"""
orderBook = self.client.get_order_book(market)
return pd.DataFrame(orderBook.bids).astype(float), pd.DataFrame(orderBook.asks).astype(float)
def get_usdc_balance(self):
"""
Get the USDC balance of the connected wallet.
Returns:
float: USDC balance in decimal format
"""
return self.usdc_contract.functions.balanceOf(self.browser_wallet).call() / 10**6
def get_pos_balance(self):
"""
Get the total value of all positions for the connected wallet.
Returns:
float: Total position value in USDC
"""
res = requests.get(f'https://data-api.polymarket.com/value?user={self.browser_wallet}')
return float(res.json()['value'])
def get_total_balance(self):
"""
Get the combined value of USDC balance and all positions.
Returns:
float: Total account value in USDC
"""
return self.get_usdc_balance() + self.get_pos_balance()
def get_all_positions(self):
"""
Get all positions for the connected wallet across all markets.
Returns:
DataFrame: All positions with details like market, size, avgPrice
"""
res = requests.get(f'https://data-api.polymarket.com/positions?user={self.browser_wallet}')
return pd.DataFrame(res.json())
def get_raw_position(self, tokenId):
"""
Get the raw token balance for a specific market outcome token.
Args:
tokenId (int): Token ID to query
Returns:
int: Raw token amount (before decimal conversion)
"""
return int(self.conditional_tokens.functions.balanceOf(self.browser_wallet, int(tokenId)).call())
def get_position(self, tokenId):
"""
Get both raw and formatted position size for a token.
Args:
tokenId (int): Token ID to query
Returns:
tuple: (raw_position, shares) - Raw token amount and decimal shares
Shares less than 1 are treated as 0 to avoid dust amounts
"""
raw_position = self.get_raw_position(tokenId)
shares = float(raw_position / 1e6)
# Ignore very small positions (dust)
if shares < 1:
shares = 0
return raw_position, shares
def get_all_orders(self):
"""
Get all open orders for the connected wallet.
Returns:
DataFrame: All open orders with their details
"""
orders_df = pd.DataFrame(self.client.get_orders())
# Convert numeric columns to float
for col in ['original_size', 'size_matched', 'price']:
if col in orders_df.columns:
orders_df[col] = orders_df[col].astype(float)
return orders_df
def get_market_orders(self, market):
"""
Get all open orders for a specific market.
Args:
market (str): Market ID to query
Returns:
DataFrame: Open orders for the specified market
"""
orders_df = pd.DataFrame(self.client.get_orders(OpenOrderParams(
market=market,
)))
# Convert numeric columns to float
for col in ['original_size', 'size_matched', 'price']:
if col in orders_df.columns:
orders_df[col] = orders_df[col].astype(float)
return orders_df
def cancel_all_asset(self, asset_id):
"""
Cancel all orders for a specific asset token.
Args:
asset_id (str): Asset token ID
"""
self.client.cancel_market_orders(asset_id=str(asset_id))
def cancel_all_market(self, marketId):
"""
Cancel all orders in a specific market.
Args:
marketId (str): Market ID
"""
self.client.cancel_market_orders(market=marketId)
def merge_positions(self, amount_to_merge, condition_id, is_neg_risk_market):
"""
Merge positions in a market to recover collateral.
This function calls the external poly_merger Node.js script to execute
the merge operation on-chain. When you hold both YES and NO positions
in the same market, merging them recovers your USDC.
Args:
amount_to_merge (int): Raw token amount to merge (before decimal conversion)
condition_id (str): Market condition ID
is_neg_risk_market (bool): Whether this is a negative risk market
Returns:
str: Transaction hash or output from the merge script
Raises:
Exception: If the merge operation fails
"""
amount_to_merge_str = str(amount_to_merge)
# Prepare the command to run the JavaScript script
node_command = f'node poly_merger/merge.js {amount_to_merge_str} {condition_id} {"true" if is_neg_risk_market else "false"}'
print(node_command)
# Run the command and capture the output
result = subprocess.run(node_command, shell=True, capture_output=True, text=True)
# Check if there was an error
if result.returncode != 0:
print("Error:", result.stderr)
raise Exception(f"Error in merging positions: {result.stderr}")
print("Done merging")
# Return the transaction hash or output
return result.stdout
poly_data/trading_utils.py
-197
View File
@@ -1,197 +0,0 @@
import math
from poly_data.data_utils import update_positions
import poly_data.global_state as global_state
# def get_avgPrice(position, assetId):
# curr_global = global_state.all_positions[global_state.all_positions['asset'] == str(assetId)]
# api_position_size = 0
# api_avgPrice = 0
# if len(curr_global) > 0:
# c_row = curr_global.iloc[0]
# api_avgPrice = round(c_row['avgPrice'], 2)
# api_position_size = c_row['size']
# if position > 0:
# if abs((api_position_size - position)/position * 100) > 5:
# print("Updating global positions")
# update_positions()
# try:
# c_row = curr_global.iloc[0]
# api_avgPrice = round(c_row['avgPrice'], 2)
# api_position_size = c_row['size']
# except:
# return 0
# return api_avgPrice
def get_best_bid_ask_deets(market, name, size, deviation_threshold=0.05):
best_bid, best_bid_size, second_best_bid, second_best_bid_size, top_bid = find_best_price_with_size(global_state.all_data[market]['bids'], size, reverse=True)
best_ask, best_ask_size, second_best_ask, second_best_ask_size, top_ask = find_best_price_with_size(global_state.all_data[market]['asks'], size, reverse=False)
# Handle None values in mid_price calculation
if best_bid is not None and best_ask is not None:
mid_price = (best_bid + best_ask) / 2
bid_sum_within_n_percent = sum(size for price, size in global_state.all_data[market]['bids'].items() if best_bid <= price <= mid_price * (1 + deviation_threshold))
ask_sum_within_n_percent = sum(size for price, size in global_state.all_data[market]['asks'].items() if mid_price * (1 - deviation_threshold) <= price <= best_ask)
else:
mid_price = None
bid_sum_within_n_percent = 0
ask_sum_within_n_percent = 0
if name == 'token2':
# Handle None values before arithmetic operations
if all(x is not None for x in [best_bid, best_ask, second_best_bid, second_best_ask, top_bid, top_ask]):
best_bid, second_best_bid, top_bid, best_ask, second_best_ask, top_ask = 1 - best_ask, 1 - second_best_ask, 1 - top_ask, 1 - best_bid, 1 - second_best_bid, 1 - top_bid
best_bid_size, second_best_bid_size, best_ask_size, second_best_ask_size = best_ask_size, second_best_ask_size, best_bid_size, second_best_bid_size
bid_sum_within_n_percent, ask_sum_within_n_percent = ask_sum_within_n_percent, bid_sum_within_n_percent
else:
# Handle case where some prices are None - use available values or defaults
if best_bid is not None and best_ask is not None:
best_bid, best_ask = 1 - best_ask, 1 - best_bid
best_bid_size, best_ask_size = best_ask_size, best_bid_size
if second_best_bid is not None:
second_best_bid = 1 - second_best_bid
if second_best_ask is not None:
second_best_ask = 1 - second_best_ask
if top_bid is not None:
top_bid = 1 - top_bid
if top_ask is not None:
top_ask = 1 - top_ask
bid_sum_within_n_percent, ask_sum_within_n_percent = ask_sum_within_n_percent, bid_sum_within_n_percent
#return as dictionary
return {
'best_bid': best_bid,
'best_bid_size': best_bid_size,
'second_best_bid': second_best_bid,
'second_best_bid_size': second_best_bid_size,
'top_bid': top_bid,
'best_ask': best_ask,
'best_ask_size': best_ask_size,
'second_best_ask': second_best_ask,
'second_best_ask_size': second_best_ask_size,
'top_ask': top_ask,
'bid_sum_within_n_percent': bid_sum_within_n_percent,
'ask_sum_within_n_percent': ask_sum_within_n_percent
}
def find_best_price_with_size(price_dict, min_size, reverse=False):
lst = list(price_dict.items())
if reverse:
lst.reverse()
best_price, best_size = None, None
second_best_price, second_best_size = None, None
top_price = None
set_best = False
for price, size in lst:
if top_price is None:
top_price = price
if set_best:
second_best_price, second_best_size = price, size
break
if size > min_size:
if best_price is None:
best_price, best_size = price, size
set_best = True
return best_price, best_size, second_best_price, second_best_size, top_price
def get_order_prices(best_bid, best_bid_size, top_bid, best_ask, best_ask_size, top_ask, avgPrice, row):
bid_price = best_bid + row['tick_size']
ask_price = best_ask - row['tick_size']
if best_bid_size < row['min_size'] * 1.5:
bid_price = best_bid
if best_ask_size < 250 * 1.5:
ask_price = best_ask
if bid_price >= top_ask:
bid_price = top_bid
if ask_price <= top_bid:
ask_price = top_ask
if bid_price == ask_price:
bid_price = top_bid
ask_price = top_ask
# if ask_price <= avgPrice:
# if avgPrice - ask_price <= (row['max_spread']*1.7/100):
# ask_price = avgPrice
#temp for sleep
if ask_price <= avgPrice and avgPrice > 0:
ask_price = avgPrice
return bid_price, ask_price
def round_down(number, decimals):
factor = 10 ** decimals
return math.floor(number * factor) / factor
def round_up(number, decimals):
factor = 10 ** decimals
return math.ceil(number * factor) / factor
def get_buy_sell_amount(position, bid_price, row, other_token_position=0):
buy_amount = 0
sell_amount = 0
# Get max_size, defaulting to trade_size if not specified
max_size = row.get('max_size', row['trade_size'])
trade_size = row['trade_size']
# Calculate total exposure across both sides
total_exposure = position + other_token_position
# If we haven't reached max_size on either side, continue building
if position < max_size:
# Continue quoting trade_size amounts until we reach max_size
remaining_to_max = max_size - position
buy_amount = min(trade_size, remaining_to_max)
# Only sell if we have substantial position (to allow for exit when needed)
if position >= trade_size:
sell_amount = min(position, trade_size)
else:
sell_amount = 0
else:
# We've reached max_size, implement progressive exit strategy
# Always offer to sell trade_size amount when at max_size
sell_amount = min(position, trade_size)
# Continue quoting to buy if total exposure warrants it
if total_exposure < max_size * 2: # Allow some flexibility for market making
buy_amount = trade_size
else:
buy_amount = 0
# Ensure minimum order size compliance
if buy_amount > 0.7 * row['min_size'] and buy_amount < row['min_size']:
buy_amount = row['min_size']
# Apply multiplier for low-priced assets
if bid_price < 0.1 and buy_amount > 0:
multiplier = row.get('multiplier', '')
if multiplier != '':
print(f"Multiplying buy amount by {int(multiplier)}")
buy_amount = buy_amount * int(multiplier)
return buy_amount, sell_amount
poly_data/utils.py
-68
View File
@@ -1,68 +0,0 @@
import json
from poly_utils.google_utils import get_spreadsheet
import pandas as pd
import os
def pretty_print(txt, dic):
print("\n", txt, json.dumps(dic, indent=4))
def get_sheet_df(read_only=None):
"""
Get sheet data with optional read-only mode
Args:
read_only (bool): If None, auto-detects based on credentials availability
"""
all = 'All Markets'
sel = 'Selected Markets'
# Auto-detect read-only mode if not specified
if read_only is None:
creds_file = 'credentials.json' if os.path.exists('credentials.json') else '../credentials.json'
read_only = not os.path.exists(creds_file)
if read_only:
print("No credentials found, using read-only mode")
try:
spreadsheet = get_spreadsheet(read_only=read_only)
except FileNotFoundError:
print("No credentials found, falling back to read-only mode")
spreadsheet = get_spreadsheet(read_only=True)
wk = spreadsheet.worksheet(sel)
df = pd.DataFrame(wk.get_all_records())
df = df[df['question'] != ""].reset_index(drop=True)
wk2 = spreadsheet.worksheet(all)
df2 = pd.DataFrame(wk2.get_all_records())
df2 = df2[df2['question'] != ""].reset_index(drop=True)
result = df.merge(df2, on='question', how='inner')
wk_p = spreadsheet.worksheet('Hyperparameters')
records = wk_p.get_all_records()
hyperparams, current_type = {}, None
for r in records:
# Update current_type only when we have a non-empty type value
# Handle both string and NaN values from pandas
type_value = r['type']
if type_value and str(type_value).strip() and str(type_value) != 'nan':
current_type = str(type_value).strip()
# Skip rows where we don't have a current_type set
if current_type:
# Convert numeric values to appropriate types
value = r['value']
try:
# Try to convert to float if it's numeric
if isinstance(value, str) and value.replace('.', '').replace('-', '').isdigit():
value = float(value)
elif isinstance(value, (int, float)):
value = float(value)
except (ValueError, TypeError):
pass # Keep as string if conversion fails
hyperparams.setdefault(current_type, {})[r['param']] = value
return result, hyperparams
poly_data/websocket_handlers.py
-98
View File
@@ -1,98 +0,0 @@
import asyncio # Asynchronous I/O
import json # JSON handling
import websockets # WebSocket client
import traceback # Exception handling
from poly_data.data_processing import process_data, process_user_data
import poly_data.global_state as global_state
async def connect_market_websocket(chunk):
"""
Connect to Polymarket's market WebSocket API and process market updates.
This function:
1. Establishes a WebSocket connection to the Polymarket API
2. Subscribes to updates for a specified list of market tokens
3. Processes incoming order book and price updates
Args:
chunk (list): List of token IDs to subscribe to
Notes:
If the connection is lost, the function will exit and the main loop will
attempt to reconnect after a short delay.
"""
uri = "wss://ws-subscriptions-clob.polymarket.com/ws/market"
async with websockets.connect(uri, ping_interval=5, ping_timeout=None) as websocket:
# Prepare and send subscription message
message = {"assets_ids": chunk}
await websocket.send(json.dumps(message))
print("\n")
print(f"Sent market subscription message: {message}")
try:
# Process incoming market data indefinitely
while True:
message = await websocket.recv()
json_data = json.loads(message)
# Process order book updates and trigger trading as needed
process_data(json_data)
except websockets.ConnectionClosed:
print("Connection closed in market websocket")
print(traceback.format_exc())
except Exception as e:
print(f"Exception in market websocket: {e}")
print(traceback.format_exc())
finally:
# Brief delay before attempting to reconnect
await asyncio.sleep(5)
async def connect_user_websocket():
"""
Connect to Polymarket's user WebSocket API and process order/trade updates.
This function:
1. Establishes a WebSocket connection to the Polymarket user API
2. Authenticates using API credentials
3. Processes incoming order and trade updates for the user
Notes:
If the connection is lost, the function will exit and the main loop will
attempt to reconnect after a short delay.
"""
uri = "wss://ws-subscriptions-clob.polymarket.com/ws/user"
async with websockets.connect(uri, ping_interval=5, ping_timeout=None) as websocket:
# Prepare authentication message with API credentials
message = {
"type": "user",
"auth": {
"apiKey": global_state.client.client.creds.api_key,
"secret": global_state.client.client.creds.api_secret,
"passphrase": global_state.client.client.creds.api_passphrase
}
}
# Send authentication message
await websocket.send(json.dumps(message))
print("\n")
print(f"Sent user subscription message")
try:
# Process incoming user data indefinitely
while True:
message = await websocket.recv()
json_data = json.loads(message)
# Process trade and order updates
process_user_data(json_data)
except websockets.ConnectionClosed:
print("Connection closed in user websocket")
print(traceback.format_exc())
except Exception as e:
print(f"Exception in user websocket: {e}")
print(traceback.format_exc())
finally:
# Brief delay before attempting to reconnect
await asyncio.sleep(5)
poly_merger/README.md
-36
View File
@@ -1,36 +0,0 @@
# Poly-Merger
A utility for merging Polymarket positions efficiently. This tool helps in consolidating opposite positions in the same market, allowing you to:
1. Reduce gas costs
2. Free up capital
3. Simplify position management
## How It Works
The merger tool interacts with Polymarket's smart contracts to combine opposite positions in binary markets. When you hold both YES and NO shares in the same market, this tool will merge them to recover your USDC.
## Usage
The merger is invoked through the main Poly-Maker bot when position merging conditions are met, but you can also use it independently:
```
node merge.js [amount_to_merge] [condition_id] [is_neg_risk_market]
```
Example:
```
node merge.js 1000000 0xasdasda true
```
This would merge 1 USDC worth of opposing positions in market 0xasdasda, which is a negative risk market. 0xasdasda should be condition_id
## Prerequisites
- Node.js
- ethers.js v5.x
- A .env file with your Polygon network private key
## Notes
This implementation is based on open-source Polymarket code but has been optimized for automated market making operations.
poly_merger/merge.js
-140
View File
@@ -1,140 +0,0 @@
/**
* Poly-Merger: Position Merging Utility for Polymarket
*
* This script handles merging of YES and NO positions in Polymarket prediction markets
* to recover collateral. It works with both regular and negative risk markets.
*
* The merger supports Gnosis Safe wallets through the safe-helpers.js utility.
*
* Usage:
* node merge.js [amountToMerge] [conditionId] [isNegRiskMarket]
*
* Example:
* node merge.js 1000000 12345 true
*/
const { ethers } = require('ethers');
const { resolve } = require('path');
const { existsSync } = require('fs');
const { signAndExecuteSafeTransaction } = require('./safe-helpers');
const { safeAbi } = require('./safeAbi');
// Load environment variables
const localEnvPath = resolve(__dirname, '.env');
const parentEnvPath = resolve(__dirname, '../.env');
const envPath = existsSync(localEnvPath) ? localEnvPath : parentEnvPath;
require('dotenv').config({ path: envPath })
// Connect to Polygon network
const provider = new ethers.providers.JsonRpcProvider("https://polygon-rpc.com");
const privateKey = process.env.PK;
const wallet = new ethers.Wallet(privateKey, provider);
// Polymarket contract addresses
const addresses = {
// Adapter contract for negative risk markets
neg_risk_adapter: '0xd91E80cF2E7be2e162c6513ceD06f1dD0dA35296',
// USDC token contract on Polygon
collateral: '0x2791Bca1f2de4661ED88A30C99A7a9449Aa84174',
// Main conditional tokens contract for prediction markets
conditional_tokens: '0x4D97DCd97eC945f40cF65F87097ACe5EA0476045'
};
// Minimal ABIs for the contracts we interact with
const negRiskAdapterAbi = [
"function mergePositions(bytes32 conditionId, uint256 amount)"
];
const conditionalTokensAbi = [
"function mergePositions(address collateralToken, bytes32 parentCollectionId, bytes32 conditionId, uint256[] partition, uint256 amount)"
];
/**
* Merges YES and NO positions in a Polymarket prediction market to recover USDC collateral.
*
* This function handles both regular and negative risk markets via different contract calls.
* It uses the Gnosis Safe wallet infrastructure for secure transaction execution.
*
* @param {string|number} amountToMerge - Raw amount of tokens to merge (typically expressed in raw units, e.g., 1000000 = 1 USDC)
* @param {string|number} conditionId - The market's condition ID
* @param {boolean} isNegRiskMarket - Whether this is a negative risk market (uses different contract)
* @returns {string} The transaction hash of the merge operation
*/
async function mergePositions(amountToMerge, conditionId, isNegRiskMarket) {
// Log parameters for debugging
console.log(amountToMerge, conditionId, isNegRiskMarket);
// Prepare transaction parameters
const nonce = await provider.getTransactionCount(wallet.address);
const gasPrice = await provider.getGasPrice();
const gasLimit = 10000000; // Set high gas limit to ensure transaction completes
let tx;
// Different contract calls for different market types
if (isNegRiskMarket) {
// For negative risk markets, use the adapter contract
const negRiskAdapter = new ethers.Contract(addresses.neg_risk_adapter, negRiskAdapterAbi, wallet);
tx = await negRiskAdapter.populateTransaction.mergePositions(conditionId, amountToMerge);
} else {
// For regular markets, use the conditional tokens contract directly
const conditionalTokens = new ethers.Contract(addresses.conditional_tokens, conditionalTokensAbi, wallet);
tx = await conditionalTokens.populateTransaction.mergePositions(
addresses.collateral, // USDC contract
ethers.constants.HashZero, // Parent collection ID (0 for top-level markets)
conditionId, // Market ID
[1, 2], // Partition (indexes of outcomes to merge)
amountToMerge // Amount to merge
);
}
// Prepare full transaction object
const transaction = {
...tx,
chainId: 137, // Polygon chain ID
gasPrice: gasPrice,
gasLimit: gasLimit,
nonce: nonce
};
// Get the Safe address from environment variables
const safeAddress = process.env.BROWSER_ADDRESS;
const safe = new ethers.Contract(safeAddress, safeAbi, wallet);
// Execute the transaction through the Safe
console.log("Signing Transaction")
const txResponse = await signAndExecuteSafeTransaction(
wallet,
safe,
transaction.to,
transaction.data,
{
gasPrice: transaction.gasPrice,
gasLimit: transaction.gasLimit
}
);
console.log("Sent transaction. Waiting for response")
const txReceipt = await txResponse.wait();
console.log("merge positions " + txReceipt.transactionHash);
return txReceipt.transactionHash;
}
// Parse command line arguments
const args = process.argv.slice(2);
// Amount of tokens to merge (in raw units, e.g., 1000000 = 1 USDC)
const amountToMerge = args[0];
// The market's condition ID
const conditionId = args[1];
// Whether this is a negative risk market (true/false)
const isNegRiskMarket = args[2] === 'true';
// Execute the merge operation and handle any errors
mergePositions(amountToMerge, conditionId, isNegRiskMarket)
.catch(error => {
console.error("Error merging positions:", error);
process.exit(1);
});
poly_merger/package-lock.json
Generated
-1339
View File
File diff suppressed because it is too large Load Diff
poly_merger/package.json
-15
View File
@@ -1,15 +0,0 @@
{
"name": "poly-merger",
"version": "1.0.0",
"description": "Position merging utility for Polymarket",
"main": "merge.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "",
"license": "MIT",
"dependencies": {
"dotenv": "^16.4.5",
"ethers": "^5.6.3"
}
}
poly_merger/safe-helpers.js
-128
View File
@@ -1,128 +0,0 @@
const { BigNumber, ethers } = require('ethers');
function joinHexData(hexData) {
return `0x${hexData
.map(hex => {
const stripped = hex.replace(/^0x/, "");
return stripped.length % 2 === 0 ? stripped : "0" + stripped;
})
.join("")}`;
}
function abiEncodePacked(...params) {
return joinHexData(
params.map(({ type, value }) => {
const encoded = ethers.utils.defaultAbiCoder.encode([type], [value]);
if (type === "bytes" || type === "string") {
const bytesLength = parseInt(encoded.slice(66, 130), 16);
return encoded.slice(130, 130 + 2 * bytesLength);
}
let typeMatch = type.match(/^(?:u?int\d*|bytes\d+|address)\[\]$/);
if (typeMatch) {
return encoded.slice(130);
}
if (type.startsWith("bytes")) {
const bytesLength = parseInt(type.slice(5));
return encoded.slice(2, 2 + 2 * bytesLength);
}
typeMatch = type.match(/^u?int(\d*)$/);
if (typeMatch) {
if (typeMatch[1] !== "") {
const bytesLength = parseInt(typeMatch[1]) / 8;
return encoded.slice(-2 * bytesLength);
}
return encoded.slice(-64);
}
if (type === "address") {
return encoded.slice(-40);
}
throw new Error(`unsupported type ${type}`);
})
);
}
async function signTransactionHash(signer, message) {
const messageArray = ethers.utils.arrayify(message);
let sig = await signer.signMessage(messageArray);
let sigV = parseInt(sig.slice(-2), 16);
switch (sigV) {
case 0:
case 1:
sigV += 31;
break;
case 27:
case 28:
sigV += 4;
break;
default:
throw new Error("Invalid signature");
}
sig = sig.slice(0, -2) + sigV.toString(16);
return {
r: BigNumber.from("0x" + sig.slice(2, 66)).toString(),
s: BigNumber.from("0x" + sig.slice(66, 130)).toString(),
v: BigNumber.from("0x" + sig.slice(130, 132)).toString(),
};
}
async function signAndExecuteSafeTransaction(signer, safe, to, data, overrides = {}) {
const nonce = await safe.nonce();
console.log("Nonce for safe: ", nonce);
const value = "0";
const safeTxGas = "0";
const baseGas = "0";
const gasPrice = "0";
const gasToken = ethers.constants.AddressZero;
const refundReceiver = ethers.constants.AddressZero;
const operation = 0;
const txHash = await safe.getTransactionHash(
to,
value,
data,
operation,
safeTxGas,
baseGas,
gasPrice,
gasToken,
refundReceiver,
nonce
);
console.log("Transaction hash: ", txHash);
const rsvSignature = await signTransactionHash(signer, txHash);
const packedSig = abiEncodePacked(
{ type: "uint256", value: rsvSignature.r },
{ type: "uint256", value: rsvSignature.s },
{ type: "uint8", value: rsvSignature.v }
);
console.log("Executing transaction");
return safe.execTransaction(
to,
value,
data,
operation,
safeTxGas,
baseGas,
gasPrice,
gasToken,
refundReceiver,
packedSig,
overrides
);
}
module.exports = {
signAndExecuteSafeTransaction,
};
poly_merger/safeAbi.js
-1140
View File
File diff suppressed because it is too large Load Diff
poly_stats/account_stats.py
-137
View File
@@ -1,137 +0,0 @@
import pandas as pd
from py_clob_client.headers.headers import create_level_2_headers
from py_clob_client.clob_types import RequestArgs
from poly_utils.google_utils import get_spreadsheet
from gspread_dataframe import set_with_dataframe
import requests
import json
import os
from dotenv import load_dotenv
load_dotenv()
spreadsheet = get_spreadsheet()
def get_markets_df(wk_full):
markets_df = pd.DataFrame(wk_full.get_all_records())
markets_df = markets_df[['question', 'answer1', 'answer2', 'token1', 'token2']]
markets_df['token1'] = markets_df['token1'].astype(str)
markets_df['token2'] = markets_df['token2'].astype(str)
return markets_df
def get_all_orders(client):
orders = client.client.get_orders()
orders_df = pd.DataFrame(orders)
if len(orders_df) > 0:
orders_df['order_size'] = orders_df['original_size'].astype('float') - orders_df['size_matched'].astype('float')
orders_df = orders_df[['asset_id', 'order_size', 'side', 'price']]
orders_df = orders_df.rename(columns={'side': 'order_side', 'price': 'order_price'})
return orders_df
else:
return pd.DataFrame()
def get_all_positions(client):
try:
positions = client.get_all_positions()
positions = positions[['asset', 'size', 'avgPrice', 'curPrice', 'percentPnl']]
positions = positions.rename(columns={'size': 'position_size'})
return positions
except:
return pd.DataFrame()
def combine_dfs(orders_df, positions, markets_df, selected_df):
merged_df = orders_df.merge(positions, left_on=['asset_id'], right_on=['asset'], how='outer')
merged_df['asset_id'] = merged_df['asset_id'].combine_first(merged_df['asset'])
merged_df = merged_df.drop(columns='asset', axis=1)
merge_token1 = merged_df.merge(markets_df, left_on='asset_id', right_on='token1', how='inner')
merge_token1['merged_with'] = 'token1'
# Merge with token2
merge_token2 = merged_df.merge(markets_df, left_on='asset_id', right_on='token2', how='inner')
merge_token2['merged_with'] = 'token2'
# Combine the results
combined_df = pd.concat([merge_token1, merge_token2])
assert len(merged_df) == len(combined_df)
combined_df['answer'] = combined_df.apply(
lambda row: row['answer1'] if row['merged_with'] == 'token1' else row['answer2'], axis=1
)
combined_df = combined_df[['question', 'answer', 'order_size', 'order_side', 'order_price', 'position_size', 'avgPrice', 'curPrice']]
combined_df['order_side'] = combined_df['order_side'].fillna('')
combined_df = combined_df.fillna(0)
combined_df['marketInSelected'] = combined_df['question'].isin(selected_df['question'])
combined_df = combined_df.sort_values('question')
combined_df = combined_df.sort_values('marketInSelected')
return combined_df
def get_earnings(client):
args = RequestArgs(method='GET', request_path='/rewards/user/markets')
l2Headers = create_level_2_headers(client.signer, client.creds, args)
url = "https://polymarket.com/api/rewards/markets"
cursor = ''
markets = []
params = {
"l2Headers": json.dumps(l2Headers),
"orderBy": "earnings",
"position": "DESC",
"makerAddress": os.getenv('BROWSER_WALLET'),
"authenticationType": "eoa",
"nextCursor": cursor,
"requestPath": "/rewards/user/markets"
}
r = requests.get(url, params=params)
results = r.json()
data = pd.DataFrame(results['data'])
data['earnings'] = data['earnings'].apply(lambda x: x[0]['earnings'])
data = data[data['earnings'] > 0].reset_index(drop=True)
data = data[['question', 'earnings', 'earning_percentage']]
return data
def update_stats_once(client):
spreadsheet = get_spreadsheet()
wk_full = spreadsheet.worksheet('Full Markets')
wk_summary = spreadsheet.worksheet('Summary')
wk_sel = spreadsheet.worksheet('Selected Markets')
selected_df = pd.DataFrame(wk_sel.get_all_records())
markets_df = get_markets_df(wk_full)
print("Got spreadsheet...")
orders_df = get_all_orders(client)
print("Got Orders...")
positions = get_all_positions(client)
print("Got Positions...")
if len(positions) > 0 or len(orders_df) > 0:
combined_df = combine_dfs(orders_df, positions, markets_df, selected_df)
earnings = get_earnings(client.client)
print("Got Earnings...")
combined_df = combined_df.merge(earnings, on='question', how='left')
combined_df = combined_df.fillna(0)
combined_df = combined_df.round(2)
combined_df = combined_df.sort_values('earnings', ascending=False)
combined_df = combined_df[['question', 'answer', 'order_size', 'position_size', 'marketInSelected', 'earnings', 'earning_percentage']]
wk_summary.clear()
set_with_dataframe(wk_summary, combined_df, include_index=False, include_column_header=True, resize=True)
else:
print("Position or order is empty")
poly_utils/google_utils.py
-155
View File
@@ -1,155 +0,0 @@
from google.oauth2.service_account import Credentials
import gspread
import os
import pandas as pd
import requests
import re
from dotenv import load_dotenv
load_dotenv()
def get_spreadsheet(read_only=False):
"""
Get Google Spreadsheet with optional read-only mode.
Args:
read_only (bool): If True, uses public CSV export when credentials are missing
Returns:
Spreadsheet object or ReadOnlySpreadsheet wrapper for read-only mode
"""
spreadsheet_url = os.getenv("SPREADSHEET_URL")
if not spreadsheet_url:
raise ValueError("SPREADSHEET_URL environment variable is not set")
# Check for credentials
creds_file = 'credentials.json' if os.path.exists('credentials.json') else '../credentials.json'
if not os.path.exists(creds_file):
if read_only:
return ReadOnlySpreadsheet(spreadsheet_url)
else:
raise FileNotFoundError(f"Credentials file not found at {creds_file}. Use read_only=True for read-only access.")
# Normal authenticated access
scope = ["https://spreadsheets.google.com/feeds", "https://www.googleapis.com/auth/drive"]
credentials = Credentials.from_service_account_file(creds_file, scopes=scope)
client = gspread.authorize(credentials)
spreadsheet = client.open_by_url(spreadsheet_url)
return spreadsheet
class ReadOnlySpreadsheet:
"""Read-only wrapper for Google Sheets using public CSV export"""
def __init__(self, spreadsheet_url):
self.spreadsheet_url = spreadsheet_url
self.sheet_id = self._extract_sheet_id(spreadsheet_url)
def _extract_sheet_id(self, url):
"""Extract sheet ID from Google Sheets URL"""
match = re.search(r'/spreadsheets/d/([a-zA-Z0-9-_]+)', url)
if not match:
raise ValueError("Invalid Google Sheets URL")
return match.group(1)
def worksheet(self, title):
"""Return a read-only worksheet"""
return ReadOnlyWorksheet(self.sheet_id, title)
class ReadOnlyWorksheet:
"""Read-only worksheet that fetches data via CSV export"""
def __init__(self, sheet_id, title):
self.sheet_id = sheet_id
self.title = title
def get_all_records(self):
"""Get all records from the worksheet as a list of dictionaries"""
try:
# URL encode the sheet title to handle spaces and special characters
import urllib.parse
encoded_title = urllib.parse.quote(self.title)
# Map known sheet names to likely GID positions
# Based on the sheet order: Full Markets, All Markets, Volatility Markets, Selected Markets, Hyperparameters
sheet_gid_mapping = {
'Full Markets': 0,
'All Markets': 1,
'Volatility Markets': 2,
'Selected Markets': 3,
'Hyperparameters': 4
}
# Try multiple URL formats for accessing the sheet
urls_to_try = [
f"https://docs.google.com/spreadsheets/d/{self.sheet_id}/gviz/tq?tqx=out:csv&sheet={encoded_title}",
f"https://docs.google.com/spreadsheets/d/{self.sheet_id}/gviz/tq?tqx=out:csv&sheet={self.title}",
]
# Add GID-based URL if we know the likely position for this sheet
if self.title in sheet_gid_mapping:
gid = sheet_gid_mapping[self.title]
urls_to_try.append(f"https://docs.google.com/spreadsheets/d/{self.sheet_id}/export?format=csv&gid={gid}")
# Also try a few common GID positions as fallback
for gid in [0, 1, 2, 3, 4]:
urls_to_try.append(f"https://docs.google.com/spreadsheets/d/{self.sheet_id}/export?format=csv&gid={gid}")
for csv_url in urls_to_try:
try:
print(f"Trying to fetch sheet '{self.title}' from: {csv_url}")
response = requests.get(csv_url, timeout=30)
response.raise_for_status()
# Read CSV data into DataFrame
from io import StringIO
df = pd.read_csv(StringIO(response.text))
# Check if we got meaningful data (not empty or error response)
if not df.empty and len(df.columns) > 1:
# For Hyperparameters sheet, verify it has the expected columns
if self.title == 'Hyperparameters':
expected_cols = ['type', 'param', 'value']
if all(col in df.columns for col in expected_cols):
print(f"Successfully fetched {len(df)} hyperparameter records")
return df.to_dict('records')
else:
print(f"Sheet doesn't match Hyperparameters format. Columns: {list(df.columns)}")
continue
else:
print(f"Successfully fetched {len(df)} records from sheet '{self.title}'")
# Convert to list of dictionaries (same format as gspread)
return df.to_dict('records')
except Exception as url_error:
print(f"Failed with URL {csv_url}: {url_error}")
continue
print(f"All URL attempts failed for sheet '{self.title}'")
return []
except Exception as e:
print(f"Warning: Could not fetch data from sheet '{self.title}': {e}")
return []
def get_all_values(self):
"""Get all values from the worksheet as a list of lists"""
try:
csv_url = f"https://docs.google.com/spreadsheets/d/{self.sheet_id}/gviz/tq?tqx=out:csv&sheet={self.title}"
response = requests.get(csv_url, timeout=30)
response.raise_for_status()
# Read CSV and return as list of lists
from io import StringIO
df = pd.read_csv(StringIO(response.text))
# Include headers and convert to list of lists
headers = [df.columns.tolist()]
data = df.values.tolist()
return headers + data
except Exception as e:
print(f"Warning: Could not fetch data from sheet '{self.title}': {e}")
return []
pyproject.toml
+59 -24
View File
@@ -1,33 +1,39 @@
[project]
name = "poly-maker"
version = "0.1.0"
description = "A market making bot for Polymarket prediction markets"
name = "polymaker"
version = "2.0.0"
description = "Maker-only market-making bot for Polymarket (CLOB V2), local-file config, political markets"
readme = "README.md"
requires-python = ">=3.9.10"
requires-python = ">=3.12"
license = { text = "MIT" }
dependencies = [
"py-clob-client==0.28.0",
"python-dotenv==1.2.1",
"pandas==2.3.3",
"gspread==6.2.1",
"gspread-dataframe==4.0.0",
"sortedcontainers==2.4.0",
"eth-account==0.13.7",
"eth-utils==5.3.1",
"poly_eip712_structs==0.0.1",
"py_order_utils==0.3.2",
"requests==2.32.5",
"websockets==15.0.1",
"cryptography==46.0.3",
"google-auth==2.42.1",
"web3==7.14.0",
"py-clob-client-v2==1.0.2",
"web3>=7.14",
"httpx>=0.28",
"websockets>=15.0",
"pydantic>=2.10",
"pydantic-settings>=2.7",
"structlog>=25.1",
"typer>=0.15",
"rich>=14.0",
"watchfiles>=1.0",
"python-dotenv>=1.0",
"sortedcontainers>=2.4",
"uvloop>=0.21 ; sys_platform != 'win32'",
"socksio>=1.0",
"python-socks>=2.4",
]
[project.scripts]
polymaker = "polymaker.cli:app"
[project.optional-dependencies]
dev = [
"black==24.4.2",
"pytest==8.2.2",
"pytest>=8.3",
"pytest-asyncio>=0.25",
"respx>=0.22",
"ruff>=0.9",
"mypy>=1.14",
]
[build-system]
@@ -35,8 +41,37 @@ requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["poly_data", "poly_stats", "poly_utils", "data_updater"]
packages = ["src/polymaker"]
[tool.black]
[tool.ruff]
line-length = 100
target-version = ["py39"]
target-version = "py312"
src = ["src", "tests"]
[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B", "SIM", "ASYNC"]
ignore = [
"E501", # line length handled by formatter
"UP042", # str+Enum is intentional (JSON-serializable, API-string valued)
"ASYNC109", # timeout params on client wrappers are fine
"SIM108", # ternary-vs-if is a readability call
]
[tool.mypy]
python_version = "3.12"
packages = ["polymaker"]
strict = true
warn_return_any = true
disallow_untyped_defs = true
# third-party libs without stubs
[[tool.mypy.overrides]]
module = ["py_clob_client_v2.*", "sortedcontainers.*", "web3.*", "eth_account.*"]
ignore_missing_imports = true
[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths = ["tests"]
filterwarnings = ["ignore::DeprecationWarning"]
[tool.uv]
package = true
src/polymaker/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""polymaker — maker-only market-making bot for Polymarket CLOB V2."""
__version__ = "2.0.0"
poly_data/__init__.py → src/polymaker/catalog/__init__.py
View File
src/polymaker/catalog/gamma.py
+187
View File
@@ -0,0 +1,187 @@
"""Async Gamma API client for market discovery.
Gamma (https://gamma-api.polymarket.com, no auth) returns everything the v1
scanner burned two extra REST calls per market to compute: best bid/ask,
liquidity, volume, reward params, fee schedule, tick size, tokens. We filter
server-side by the politics tag and liquidity/volume, so a full political-market
sweep is a handful of paginated requests.
"""
from __future__ import annotations
import json
from collections.abc import AsyncIterator
from typing import Any
import httpx
from polymaker.domain import MarketMeta, TokenMeta
from polymaker.logging import get_logger
log = get_logger("catalog.gamma")
POLITICS_TAG_SLUG = "politics"
class GammaClient:
"""Thin async wrapper over the Gamma REST endpoints we use."""
def __init__(self, host: str = "https://gamma-api.polymarket.com", timeout: float = 20.0) -> None:
self._host = host.rstrip("/")
self._client = httpx.AsyncClient(base_url=self._host, timeout=timeout)
async def __aenter__(self) -> GammaClient:
return self
async def __aexit__(self, *exc: object) -> None:
await self.aclose()
async def aclose(self) -> None:
await self._client.aclose()
async def resolve_tag_id(self, slug: str) -> str | None:
try:
r = await self._client.get(f"/tags/slug/{slug}")
r.raise_for_status()
return str(r.json()["id"])
except (httpx.HTTPError, KeyError, json.JSONDecodeError):
log.warning("tag_resolve_failed", slug=slug)
return None
async def iter_markets(
self,
*,
tag_id: str | None = None,
related_tags: bool = True,
min_liquidity: float = 0.0,
min_volume_24hr: float = 0.0,
limit: int = 100, # Gamma caps a page at 100 regardless of a higher ask
max_pages: int = 25,
) -> AsyncIterator[dict[str, Any]]:
"""Yield raw active/open market dicts, offset-paginated.
Uses the offset `/markets` endpoint because it reliably supports
`tag_id` filtering today. (Keyset is the go-forward per docs; switch when
it supports tag filtering. See docs/scoping/03-api-layer.md §4.)
"""
offset = 0
for _ in range(max_pages):
params: dict[str, Any] = {
"limit": limit,
"offset": offset,
"active": "true",
"closed": "false",
"order": "volume24hr",
"ascending": "false",
}
if tag_id:
params["tag_id"] = tag_id
params["related_tags"] = "true" if related_tags else "false"
if min_liquidity > 0:
params["liquidity_num_min"] = min_liquidity
if min_volume_24hr > 0:
params["volume_num_min"] = min_volume_24hr
r = await self._client.get("/markets", params=params)
r.raise_for_status()
batch = r.json()
if not batch:
return
for m in batch:
yield m
if len(batch) < limit:
return
offset += limit
def parse_market(raw: dict[str, Any], reward_rates: dict[str, float] | None = None) -> MarketMeta | None:
"""Convert a Gamma market dict into our MarketMeta, or None if unusable."""
try:
if not raw.get("acceptingOrders", False):
return None
token_ids = _json_list(raw.get("clobTokenIds"))
outcomes = _json_list(raw.get("outcomes"))
if len(token_ids) != 2 or len(outcomes) != 2:
return None # only binary markets
condition_id = raw["conditionId"]
rate_map = reward_rates or {}
fee = raw.get("feeSchedule") or {}
taker_rate = float(fee.get("rate", 0.0) or 0.0)
event_id = None
events = raw.get("events") or []
if events:
event_id = str(events[0].get("id")) if events[0].get("id") is not None else None
return MarketMeta(
condition_id=condition_id,
question=raw.get("question", ""),
slug=raw.get("slug", ""),
tokens=(
TokenMeta(str(token_ids[0]), str(outcomes[0])),
TokenMeta(str(token_ids[1]), str(outcomes[1])),
),
tick_size=float(raw.get("orderPriceMinTickSize", 0.001) or 0.001),
neg_risk=bool(raw.get("negRisk", False)),
min_order_size=float(raw.get("orderMinSize", 5) or 5),
rewards_min_size=float(raw.get("rewardsMinSize", 0) or 0),
rewards_max_spread=float(raw.get("rewardsMaxSpread", 0) or 0),
rewards_daily_rate=float(rate_map.get(condition_id, 0.0)),
maker_fee_bps=0, # V2: makers pay zero
taker_fee_bps=int(round(taker_rate * 10000)),
fees_enabled=bool(raw.get("feesEnabled", False)),
rebate_rate=float(fee.get("rebateRate", 0.0) or 0.0),
end_date_iso=raw.get("endDate"),
event_id=event_id,
best_bid=float(raw.get("bestBid", 0) or 0),
best_ask=float(raw.get("bestAsk", 0) or 0),
liquidity_num=float(raw.get("liquidityNum", 0) or 0),
volume_num=float(raw.get("volumeNum", 0) or 0),
)
except (KeyError, ValueError, TypeError) as exc:
log.warning("parse_market_failed", err=str(exc), slug=raw.get("slug"))
return None
def _json_list(value: Any) -> list[Any]:
"""clobTokenIds / outcomes arrive as JSON-encoded strings."""
if value is None:
return []
if isinstance(value, list):
return value
try:
parsed = json.loads(value)
return parsed if isinstance(parsed, list) else []
except (json.JSONDecodeError, TypeError):
return []
async def fetch_reward_rates(
clob_host: str = "https://clob.polymarket.com", timeout: float = 20.0
) -> dict[str, float]:
"""Build {condition_id: daily USDC reward rate} from CLOB sampling-markets.
These are the rewards-enabled markets; the daily rate isn't on Gamma.
"""
usdc = "0x2791bca1f2de4661ed88a30c99a7a9449aa84174"
rates: dict[str, float] = {}
async with httpx.AsyncClient(base_url=clob_host.rstrip("/"), timeout=timeout) as client:
cursor = ""
for _ in range(50):
r = await client.get("/sampling-markets", params={"next_cursor": cursor})
r.raise_for_status()
data = r.json()
for m in data.get("data", []):
cid = m.get("condition_id")
rate = 0.0
for ri in (m.get("rewards") or {}).get("rates") or []:
if str(ri.get("asset_address", "")).lower() == usdc:
rate = float(ri.get("rewards_daily_rate", 0) or 0)
break
if cid:
rates[cid] = rate
cursor = data.get("next_cursor") or ""
if not cursor or cursor == "LTE=": # "LTE=" is the documented end sentinel
break
return rates
src/polymaker/catalog/scanner.py
+63
View File
@@ -0,0 +1,63 @@
"""The scanner: sweep Gamma for political markets, score, persist to SQLite.
Replaces the v1 data_updater (hour-long crawl of every order book, written to
Google Sheets). A politics-filtered sweep here is seconds and one process.
"""
from __future__ import annotations
from dataclasses import dataclass
from polymaker.catalog.gamma import (
POLITICS_TAG_SLUG,
GammaClient,
fetch_reward_rates,
parse_market,
)
from polymaker.catalog.scoring import score_market
from polymaker.catalog.store import CatalogStore
from polymaker.domain import MarketMeta
from polymaker.logging import get_logger
log = get_logger("catalog.scanner")
@dataclass(frozen=True, slots=True)
class ScanConfig:
tag_slug: str = POLITICS_TAG_SLUG
min_liquidity: float = 1000.0
min_volume_24hr: float = 0.0
rewards_only: bool = True # keep only markets in the liquidity-rewards program
gamma_host: str = "https://gamma-api.polymarket.com"
clob_host: str = "https://clob.polymarket.com"
async def run_scan(store: CatalogStore, cfg: ScanConfig) -> list[MarketMeta]:
"""Fetch, parse, filter, score, and persist. Returns the kept markets."""
reward_rates = await fetch_reward_rates(cfg.clob_host)
log.info("reward_rates_loaded", n=len(reward_rates))
kept: list[MarketMeta] = []
async with GammaClient(cfg.gamma_host) as gamma:
tag_id = store.cached_tag(cfg.tag_slug) or await gamma.resolve_tag_id(cfg.tag_slug)
if tag_id:
store.cache_tag(cfg.tag_slug, tag_id)
seen = 0
async for raw in gamma.iter_markets(
tag_id=tag_id,
min_liquidity=cfg.min_liquidity,
min_volume_24hr=cfg.min_volume_24hr,
):
seen += 1
meta = parse_market(raw, reward_rates)
if meta is None:
continue
if cfg.rewards_only and meta.rewards_daily_rate <= 0:
continue
kept.append(meta)
for m in kept:
store.upsert_market(m, score_market(m))
log.info("scan_complete", seen=seen, kept=len(kept), tag=cfg.tag_slug)
return kept
src/polymaker/catalog/scoring.py
+80
View File
@@ -0,0 +1,80 @@
"""Market attractiveness scoring for the scanner.
Combines the v1 reward-density intuition with the new maker-rebate income
stream and penalizes spread/extremes. Higher score = more attractive to make.
Pure functions over MarketMeta.
"""
from __future__ import annotations
from dataclasses import dataclass
from polymaker.domain import MarketMeta
@dataclass(frozen=True, slots=True)
class MarketScore:
condition_id: str
reward_density: float # est. reward $/day per $100 of two-sided liquidity
rebate_potential: float # est. daily rebate $ available to makers
spread: float
extremity: float # 0 = mid ~0.5 (good), 1 = near 0/1 (bad payoff asymmetry)
score: float
def _mid(m: MarketMeta) -> float:
if m.best_bid > 0 and m.best_ask > 0:
return (m.best_bid + m.best_ask) / 2.0
return 0.5
def reward_density(m: MarketMeta, quote_size_usdc: float = 100.0) -> float:
"""Rough reward $/day if we hold ~quote_size two-sided in-band.
The exact per-order S((v-s)/v)^2 scoring depends on live competition; for
ranking we use daily_rate scaled by how much of the (small) market our
typical size represents, capped. This mirrors v1's gm_reward_per_100 as a
relative ranking signal, not an absolute forecast.
"""
if m.rewards_daily_rate <= 0 or m.rewards_max_spread <= 0:
return 0.0
liq = max(m.liquidity_num, quote_size_usdc)
our_share = min(1.0, quote_size_usdc / liq)
return m.rewards_daily_rate * our_share
def rebate_potential(m: MarketMeta) -> float:
"""Est. daily maker-rebate pool: taker_fee_rate * rebate_rate * daily volume."""
if not m.fees_enabled or m.rebate_rate <= 0 or m.taker_fee_bps <= 0:
return 0.0
daily_vol = m.volume_num # best proxy available from catalog; refined live
taker_rate = m.taker_fee_bps / 10000.0
# taker fee peaks at p*(1-p); use mid as the representative point
mid = _mid(m)
fee_factor = mid * (1.0 - mid)
return daily_vol * taker_rate * fee_factor * m.rebate_rate * 0.01 # 1% daily-vol proxy
def extremity(m: MarketMeta) -> float:
"""0 near 0.5 (balanced), ->1 near the 0/1 boundary (skip these)."""
mid = _mid(m)
return min(1.0, abs(mid - 0.5) / 0.5)
def score_market(m: MarketMeta) -> MarketScore:
rd = reward_density(m)
rp = rebate_potential(m)
ext = extremity(m)
spread = max(0.0, m.best_ask - m.best_bid) if (m.best_bid and m.best_ask) else 1.0
# income terms are additive; extremity and wide spreads discount the score
income = rd + rp
penalty = (1.0 - 0.5 * ext) * (1.0 / (1.0 + spread * 20.0))
return MarketScore(
condition_id=m.condition_id,
reward_density=round(rd, 3),
rebate_potential=round(rp, 3),
spread=round(spread, 4),
extremity=round(ext, 3),
score=round(income * penalty, 4),
)
src/polymaker/catalog/store.py
+123
View File
@@ -0,0 +1,123 @@
"""SQLite persistence for the market catalog and scan results.
Replaces the v1 "All Markets" / "Volatility Markets" Google Sheets. One local
file (state.db), queryable by the CLI. WAL mode so the running bot and a
`polymaker markets` query don't block each other.
"""
from __future__ import annotations
import json
import sqlite3
import time
from dataclasses import asdict
from pathlib import Path
from polymaker.catalog.scoring import MarketScore, score_market
from polymaker.domain import MarketMeta, TokenMeta
_SCHEMA = """
CREATE TABLE IF NOT EXISTS markets (
condition_id TEXT PRIMARY KEY,
question TEXT,
slug TEXT,
meta_json TEXT NOT NULL,
score REAL DEFAULT 0,
score_json TEXT,
scanned_ts REAL NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_markets_score ON markets(score DESC);
CREATE INDEX IF NOT EXISTS idx_markets_slug ON markets(slug);
CREATE TABLE IF NOT EXISTS tags (
slug TEXT PRIMARY KEY,
tag_id TEXT NOT NULL,
ts REAL NOT NULL
);
"""
class CatalogStore:
"""Owns the markets/tags tables in state.db."""
def __init__(self, db_path: str | Path = "state.db") -> None:
self.path = str(db_path)
self._conn = sqlite3.connect(self.path)
self._conn.row_factory = sqlite3.Row
self._conn.execute("PRAGMA journal_mode=WAL")
self._conn.executescript(_SCHEMA)
self._conn.commit()
def close(self) -> None:
self._conn.close()
def upsert_market(self, meta: MarketMeta, score: MarketScore | None = None) -> None:
sc = score or score_market(meta)
self._conn.execute(
"""INSERT INTO markets(condition_id, question, slug, meta_json, score, score_json, scanned_ts)
VALUES(?,?,?,?,?,?,?)
ON CONFLICT(condition_id) DO UPDATE SET
question=excluded.question, slug=excluded.slug, meta_json=excluded.meta_json,
score=excluded.score, score_json=excluded.score_json, scanned_ts=excluded.scanned_ts""",
(
meta.condition_id,
meta.question,
meta.slug,
_dump_meta(meta),
sc.score,
json.dumps(asdict(sc)),
time.time(),
),
)
self._conn.commit()
def upsert_many(self, metas: list[MarketMeta]) -> int:
for m in metas:
self.upsert_market(m)
return len(metas)
def get(self, condition_id: str) -> MarketMeta | None:
row = self._conn.execute(
"SELECT meta_json FROM markets WHERE condition_id=?", (condition_id,)
).fetchone()
return _load_meta(row["meta_json"]) if row else None
def get_by_slug(self, slug: str) -> MarketMeta | None:
row = self._conn.execute(
"SELECT meta_json FROM markets WHERE slug=?", (slug,)
).fetchone()
return _load_meta(row["meta_json"]) if row else None
def top(self, limit: int = 50) -> list[tuple[MarketMeta, MarketScore]]:
rows = self._conn.execute(
"SELECT meta_json, score_json FROM markets ORDER BY score DESC LIMIT ?", (limit,)
).fetchall()
out = []
for row in rows:
meta = _load_meta(row["meta_json"])
sc = MarketScore(**json.loads(row["score_json"])) if row["score_json"] else score_market(meta)
out.append((meta, sc))
return out
def cache_tag(self, slug: str, tag_id: str) -> None:
self._conn.execute(
"INSERT OR REPLACE INTO tags(slug, tag_id, ts) VALUES(?,?,?)",
(slug, tag_id, time.time()),
)
self._conn.commit()
def cached_tag(self, slug: str) -> str | None:
row = self._conn.execute("SELECT tag_id FROM tags WHERE slug=?", (slug,)).fetchone()
return row["tag_id"] if row else None
def _dump_meta(meta: MarketMeta) -> str:
d = asdict(meta)
d["tokens"] = [asdict(t) for t in meta.tokens]
return json.dumps(d)
def _load_meta(blob: str) -> MarketMeta:
d = json.loads(blob)
d["tokens"] = tuple(TokenMeta(**t) for t in d["tokens"])
return MarketMeta(**d)
src/polymaker/cli.py
+213
View File
@@ -0,0 +1,213 @@
"""polymaker command-line interface.
polymaker scan sweep Gamma for political markets -> SQLite
polymaker markets rank/browse the catalog
polymaker markets-add <slug> append a market to config/markets.toml
polymaker status positions / open orders / PnL (reads SQLite)
polymaker doctor preflight: wallet auth, balances, WS reachability
polymaker run [--paper] start the market maker
polymaker cancel-all panic button
"""
from __future__ import annotations
import asyncio
from pathlib import Path
from typing import Any
import typer
from rich.console import Console
from rich.table import Table
from polymaker import __version__
from polymaker.config import Config
app = typer.Typer(
name="polymaker",
help="Maker-only market maker for Polymarket CLOB V2.",
no_args_is_help=True,
add_completion=False,
)
console = Console()
@app.command()
def version() -> None:
"""Print the polymaker version."""
console.print(f"polymaker {__version__}")
@app.command()
def scan(
config_dir: str = typer.Option("config", help="config directory"),
min_liquidity: float = typer.Option(1000.0, help="minimum market liquidity (USDC)"),
all_markets: bool = typer.Option(False, "--all", help="include non-rewards markets"),
) -> None:
"""Sweep Gamma for political markets, score, and persist to SQLite."""
from polymaker.catalog.scanner import ScanConfig, run_scan
from polymaker.catalog.store import CatalogStore
cfg = Config.load(config_dir)
store = CatalogStore(cfg.paths.db)
async def _go() -> int:
metas = await run_scan(store, ScanConfig(min_liquidity=min_liquidity, rewards_only=not all_markets))
return len(metas)
n = asyncio.run(_go())
console.print(f"[green]Scanned and stored {n} markets.[/green] Run [bold]polymaker markets[/bold] to browse.")
store.close()
@app.command()
def markets(
config_dir: str = typer.Option("config", help="config directory"),
limit: int = typer.Option(25, help="rows to show"),
) -> None:
"""Show the top scored markets from the catalog."""
from polymaker.catalog.store import CatalogStore
cfg = Config.load(config_dir)
store = CatalogStore(cfg.paths.db)
rows = store.top(limit)
if not rows:
console.print("[yellow]Catalog empty. Run `polymaker scan` first.[/yellow]")
raise typer.Exit()
table = Table(title="Political markets by score")
for col in ("score", "reward/day", "rebate/day", "spread", "tick", "neg", "question"):
table.add_column(col, justify="right" if col != "question" else "left")
for meta, sc in rows:
table.add_row(
f"{sc.score:.2f}", f"{meta.rewards_daily_rate:.0f}", f"{sc.rebate_potential:.0f}",
f"{sc.spread:.3f}", f"{meta.tick_size:g}", "Y" if meta.neg_risk else "-",
meta.question[:60],
)
console.print(table)
console.print("\nAdd one with: [bold]polymaker markets-add <slug>[/bold] (slugs are in the catalog)")
@app.command(name="markets-add")
def markets_add(
slug: str,
profile: str = typer.Option("political-longdated", help="strategy profile"),
config_dir: str = typer.Option("config", help="config directory"),
) -> None:
"""Append a market (by slug) to config/markets.toml."""
from polymaker.catalog.store import CatalogStore
cfg = Config.load(config_dir)
store = CatalogStore(cfg.paths.db)
meta = store.get_by_slug(slug)
store.close()
if meta is None:
console.print(f"[red]No market with slug {slug!r} in the catalog. Run `polymaker scan`.[/red]")
raise typer.Exit(1)
path = Path(config_dir) / "markets.toml"
block = f'\n[[markets]]\nslug = "{slug}"\nprofile = "{profile}"\nenabled = true\n'
with path.open("a") as fh:
fh.write(block)
console.print(f"[green]Added[/green] {meta.question[:60]!r} to {path}")
@app.command()
def status(config_dir: str = typer.Option("config", help="config directory")) -> None:
"""Show positions, open orders, and marks from the local state DB."""
from polymaker.state.store import StateStore
cfg = Config.load(config_dir)
store = StateStore(cfg.paths.db)
snap = store.snapshot()
console.print(f"[bold]Open orders:[/bold] {snap['open_orders']}")
positions: dict[str, Any] = snap["positions"] # type: ignore[assignment]
if not positions:
console.print("[dim]No open positions.[/dim]")
else:
table = Table(title="Positions")
table.add_column("token")
table.add_column("size", justify="right")
table.add_column("avg", justify="right")
for tok, p in positions.items():
table.add_row(tok[:16] + "…", f"{p['size']:.2f}", f"{p['avg_price']:.3f}")
console.print(table)
store.close()
@app.command()
def doctor(config_dir: str = typer.Option("config", help="config directory")) -> None:
"""Preflight checks: config, wallet auth, balance/allowance, WS reachability."""
from polymaker.doctor import run_doctor
cfg = Config.load(config_dir)
ok = asyncio.run(run_doctor(cfg, console))
raise typer.Exit(0 if ok else 1)
@app.command()
def run(
config_dir: str = typer.Option("config", help="config directory"),
paper: bool = typer.Option(False, "--paper", help="paper mode: full pipeline, no orders posted"),
) -> None:
"""Start the market maker."""
from polymaker.engine import Engine
from polymaker.logging import configure
cfg = Config.load(config_dir)
configure(json_file=Path(cfg.paths.log_dir) / ("paper.jsonl" if paper else "live.jsonl"))
if cfg.engine.loop == "uvloop":
try:
import uvloop
uvloop.install()
except Exception: # noqa: BLE001
pass
engine = Engine(cfg, paper=paper)
async def _go() -> None:
try:
await engine.run_forever()
except (KeyboardInterrupt, asyncio.CancelledError):
pass
finally:
await engine.shutdown()
console.print(f"[bold green]Starting polymaker[/bold green] ({'PAPER' if paper else 'LIVE'})…")
try:
asyncio.run(_go())
except KeyboardInterrupt:
console.print("\n[yellow]Stopped.[/yellow]")
@app.command()
def livetest(
config_dir: str = typer.Option("config", help="config directory"),
notional: float = typer.Option(5.0, help="order notional in USDC"),
) -> None:
"""Live wallet round-trip: place a deep post-only order and cancel it (~$5)."""
from polymaker.livetest import run_livetest
cfg = Config.load(config_dir)
ok = asyncio.run(run_livetest(cfg, console, notional))
raise typer.Exit(0 if ok else 1)
@app.command(name="cancel-all")
def cancel_all(config_dir: str = typer.Option("config", help="config directory")) -> None:
"""Cancel all open orders for the wallet (panic button)."""
from polymaker.execution.gateway import ExecutionGateway
cfg = Config.load(config_dir)
gw = ExecutionGateway(cfg)
async def _go() -> None:
await gw.connect()
await gw.cancel_all()
asyncio.run(_go())
console.print("[green]Sent cancel-all.[/green]")
if __name__ == "__main__":
app()
src/polymaker/config.py
+226
View File
@@ -0,0 +1,226 @@
"""Configuration: pydantic models over local TOML files + .env secrets.
Replaces the v1 Google Sheets config entirely. Three files under config/:
config.toml engine/wallet/risk/execution settings
strategy.toml named parameter profiles
markets.toml the trade list (market -> profile + overrides)
Secrets (private key, wallet address) come only from the environment / .env.
"""
from __future__ import annotations
import os
import tomllib
from pathlib import Path
from typing import Any
from dotenv import load_dotenv
from pydantic import BaseModel, ConfigDict, Field, model_validator
from pydantic_settings import BaseSettings, SettingsConfigDict
class WalletConfig(BaseModel):
chain_id: int = 137
signature_type: int = 2
clob_host: str = "https://clob.polymarket.com"
gamma_host: str = "https://gamma-api.polymarket.com"
data_api_host: str = "https://data-api.polymarket.com"
polygon_rpc: str = "https://polygon-rpc.com"
class EngineConfig(BaseModel):
debounce_ms: int = 200
reconcile_interval_s: float = 30.0
catalog_refresh_s: float = 900.0
heartbeat: bool = True
heartbeat_interval_s: float = 5.0
journal: bool = True
loop: str = "uvloop"
class RiskConfig(BaseModel):
max_total_exposure_usdc: float = 5000.0
max_event_group_loss_usdc: float = 1000.0
max_market_notional_usdc: float = 800.0
daily_loss_kill_usdc: float = 250.0
ws_stale_halt_s: float = 10.0
max_order_error_rate: float = 0.25
class ExecutionConfig(BaseModel):
rate_budget_fraction: float = 0.25
post_only: bool = True
max_orders_per_batch: int = 15
class PathsConfig(BaseModel):
db: str = "state.db"
journal_dir: str = "journal"
log_dir: str = "logs"
class StrategyProfile(BaseModel):
"""One named parameter set. Every knob the quoter uses lives here."""
model_config = ConfigDict(extra="forbid")
# fair value
micro_levels: int = 3
flow_ewma_halflife_s: float = 120.0
# spread / skew
gamma: float = 0.5
delta_min_ticks: int = 2
c_vol: float = 1.2
c_tox: float = 2.0
# vol horizons
vol_short_halflife_s: float = 10.0
vol_long_halflife_s: float = 900.0
# sizing / inventory
base_size_usdc: float = 50.0
q_max_usdc: float = 500.0
q_soft_frac: float = 0.6
layers: int = 2
layer_step_ticks: int = 2
# placement / churn
reprice_ticks: int = 2
resize_frac: float = 0.15
min_edge_ticks: int = 1
# regime
event_cooloff_s: float = 60.0
event_jump_ticks: int = 8
event_sweep_levels: int = 3
trend_flow_z: float = 1.5
# lifecycle
end_date_taper_days: float = 7.0
reduce_only_hours: float = 24.0
halt_before_hours: float = 2.0
# exits
exit_urgency_s: float = 900.0
merge_min_size: float = 20.0
def with_overrides(self, overrides: dict[str, Any]) -> StrategyProfile:
"""Return a copy with per-market override values applied."""
if not overrides:
return self
data = self.model_dump()
for k, v in overrides.items():
if k in data:
data[k] = v
return StrategyProfile(**data)
# Keys allowed on a market entry that are NOT profile overrides.
_MARKET_RESERVED = {"slug", "condition_id", "profile", "enabled"}
class MarketEntry(BaseModel):
"""One line of the trade list. Extra keys are treated as profile overrides."""
model_config = ConfigDict(extra="allow")
slug: str | None = None
condition_id: str | None = None
profile: str = "political-longdated"
enabled: bool = True
@model_validator(mode="after")
def _need_identifier(self) -> MarketEntry:
if not self.slug and not self.condition_id:
raise ValueError("market entry needs a slug or condition_id")
return self
@property
def overrides(self) -> dict[str, Any]:
extra = self.model_extra or {}
return {k: v for k, v in extra.items() if k not in _MARKET_RESERVED}
@property
def ref(self) -> str:
return self.slug or self.condition_id or "?"
class Secrets(BaseSettings):
"""Loaded from environment / .env. Never written to disk by us."""
model_config = SettingsConfigDict(env_file=".env", extra="ignore")
pk: str = Field(default="", alias="PK")
browser_address: str = Field(default="", alias="BROWSER_ADDRESS")
polygon_rpc: str | None = Field(default=None, alias="POLYGON_RPC")
alert_webhook_url: str | None = Field(default=None, alias="ALERT_WEBHOOK_URL")
@property
def has_wallet(self) -> bool:
return bool(self.pk and self.browser_address)
class Config(BaseModel):
"""Fully-resolved configuration tree."""
wallet: WalletConfig = WalletConfig()
engine: EngineConfig = EngineConfig()
risk: RiskConfig = RiskConfig()
execution: ExecutionConfig = ExecutionConfig()
paths: PathsConfig = PathsConfig()
profiles: dict[str, StrategyProfile] = {}
markets: list[MarketEntry] = []
secrets: Secrets = Field(default_factory=Secrets)
config_dir: Path = Path("config")
@property
def proxy(self) -> str | None:
# Standard proxy env var; ALL_PROXY lets you route through an SSH tunnel
# (e.g. simulate colocation during local testing). httpx and web3 honor
# it automatically once load_dotenv() has run.
return os.environ.get("ALL_PROXY") or os.environ.get("HTTPS_PROXY")
@property
def enabled_markets(self) -> list[MarketEntry]:
return [m for m in self.markets if m.enabled]
def profile_for(self, entry: MarketEntry) -> StrategyProfile:
base = self.profiles.get(entry.profile)
if base is None:
raise KeyError(f"unknown strategy profile: {entry.profile!r}")
return base.with_overrides(entry.overrides)
@classmethod
def load(cls, config_dir: str | Path = "config", *, load_env: bool = True) -> Config:
cdir = Path(config_dir)
if load_env:
load_dotenv()
main = _read_toml(cdir / "config.toml")
strat = _read_toml(cdir / "strategy.toml")
mkts = _read_toml(cdir / "markets.toml")
profiles = {
name: StrategyProfile(**params)
for name, params in (strat.get("profiles") or {}).items()
}
markets = [MarketEntry(**m) for m in (mkts.get("markets") or [])]
return cls(
wallet=WalletConfig(**main.get("wallet", {})),
engine=EngineConfig(**main.get("engine", {})),
risk=RiskConfig(**main.get("risk", {})),
execution=ExecutionConfig(**main.get("execution", {})),
paths=PathsConfig(**main.get("paths", {})),
profiles=profiles,
markets=markets,
secrets=Secrets(),
config_dir=cdir,
)
def reload_markets(self) -> Config:
"""Re-read markets.toml only (used by the hot-reload path)."""
mkts = _read_toml(self.config_dir / "markets.toml")
self.markets = [MarketEntry(**m) for m in (mkts.get("markets") or [])]
return self
def _read_toml(path: Path) -> dict[str, Any]:
if not path.exists():
return {}
with path.open("rb") as fh:
return tomllib.load(fh)
src/polymaker/doctor.py
+184
View File
@@ -0,0 +1,184 @@
"""Preflight checks for `polymaker doctor`.
Verifies the environment is ready to trade WITHOUT posting any order:
config + secrets, CLOB/Gamma reachable, wallet auth (L1->L2 creds), collateral
balance + positions ON THE FUNDER (deposit/developer wallet, where funds live),
a live market-WS book frame, and an authenticated user-WS connection. This is
the gate before the live $5 round-trip (docs/scoping/06 Phase 2).
"""
from __future__ import annotations
import asyncio
import json
from typing import Any
import httpx
import websockets
from rich.console import Console
from polymaker.config import Config
MARKET_WS = "wss://ws-subscriptions-clob.polymarket.com/ws/market"
USER_WS = "wss://ws-subscriptions-clob.polymarket.com/ws/user"
async def run_doctor(cfg: Config, console: Console) -> bool:
ok = True
def check(label: str, passed: bool, detail: str = "") -> None:
nonlocal ok
mark = "[green]✓[/green]" if passed else "[red]✗[/red]"
console.print(f" {mark} {label}" + (f" [dim]{detail}[/dim]" if detail else ""))
ok = ok and passed
console.print("[bold]polymaker doctor[/bold]")
# ── config + secrets ────────────────────────────────────────────────
check("config loads", True, f"{len(cfg.profiles)} profiles, {len(cfg.markets)} markets")
check("PK + BROWSER_ADDRESS set", cfg.secrets.has_wallet,
"put them in .env" if not cfg.secrets.has_wallet
else f"funder {cfg.secrets.browser_address[:10]}…, sig_type {cfg.wallet.signature_type}")
# ── REST reachability ───────────────────────────────────────────────
async with httpx.AsyncClient(timeout=10) as c:
try:
r = await c.get(f"{cfg.wallet.clob_host}/ok")
check("CLOB reachable", r.status_code == 200, cfg.wallet.clob_host)
except httpx.HTTPError as e:
check("CLOB reachable", False, str(e))
try:
r = await c.get(f"{cfg.wallet.gamma_host}/markets", params={"limit": 1})
check("Gamma reachable", r.status_code == 200, cfg.wallet.gamma_host)
except httpx.HTTPError as e:
check("Gamma reachable", False, str(e))
# ── wallet auth + balance + positions (on the FUNDER) ───────────────
creds: Any = None
funder = ""
held_tokens: list[str] = []
if cfg.secrets.has_wallet:
try:
from polymaker.execution.gateway import ExecutionGateway
gw = ExecutionGateway(cfg)
await gw.connect()
creds = gw.creds
funder = gw.funder
check("wallet auth (L2 creds derived)", bool(gw.creds),
f"signer {gw.address[:10]}… signs for funder {funder[:10]}…")
ba = await gw.balance_allowance()
bal = _extract_balance(ba)
check("collateral (pUSD) balance readable", bal is not None,
f"≈{bal:.2f} pUSD on funder {funder[:10]}…" if bal is not None else "check allowances")
if bal is not None and bal <= 0:
console.print(" [yellow]! balance is 0 — deposit USDC (mints pUSD) and set "
"allowances from the deposit wallet (trade once in the UI)[/yellow]")
positions = await gw.positions()
held_tokens = list(positions)
total_shares = sum(sz for sz, _ in positions.values())
check("positions readable (on funder)", True,
f"{len(positions)} positions, {total_shares:.0f} shares total")
except Exception as e: # noqa: BLE001
check("wallet auth (L2 creds derived)", False, str(e))
console.print(" [yellow]! signature-type mismatch? deposit wallets use sig_type=3 "
"(config.toml). See docs 03 §9.[/yellow]")
else:
console.print(" [yellow]! skipping wallet checks (no secrets)[/yellow]")
if cfg.proxy:
console.print(f" [dim]· routing via proxy {cfg.proxy.split('@')[-1]}[/dim]")
# ── live market WS: receive an actual book frame ────────────────────
token = held_tokens[0] if held_tokens else await _top_political_token(cfg)
if token:
passed, detail = await _market_ws_book(token, cfg.proxy)
check("market WS live book frame", passed, detail)
else:
check("market WS live book frame", False, "no token to subscribe to")
# ── live user WS: authenticate ──────────────────────────────────────
if creds is not None:
markets = [cfg.markets[0].condition_id] if cfg.markets and cfg.markets[0].condition_id else []
passed, detail = await _user_ws_auth(creds, markets, cfg.proxy)
check("user WS authenticated", passed, detail)
else:
console.print(" [dim]· skipping user WS (needs wallet creds)[/dim]")
console.print(f"\n[bold]{'READY' if ok else 'NOT READY'}[/bold]")
return ok
async def _market_ws_book(token: str, proxy: str | None = None) -> tuple[bool, str]:
"""Subscribe to a token and confirm a real `book` frame arrives."""
kw: dict[str, Any] = {"ping_interval": 5, "ping_timeout": None, "open_timeout": 10}
if proxy:
kw["proxy"] = proxy
try:
async with websockets.connect(MARKET_WS, **kw) as ws:
await ws.send(json.dumps({"assets_ids": [token], "type": "market"}))
for _ in range(12):
raw = await asyncio.wait_for(ws.recv(), timeout=8)
data = json.loads(raw)
for m in data if isinstance(data, list) else [data]:
if isinstance(m, dict) and m.get("event_type") == "book":
nb, na = len(m.get("bids", [])), len(m.get("asks", []))
return True, f"book received: {nb} bids / {na} asks"
except Exception as e: # noqa: BLE001
return False, str(e)[:80]
return False, "no book frame within timeout"
async def _user_ws_auth(creds: Any, markets: list[str], proxy: str | None = None) -> tuple[bool, str]:
"""Authenticate on the user channel and confirm the server accepts it."""
kw: dict[str, Any] = {"ping_interval": 5, "ping_timeout": None, "open_timeout": 10}
if proxy:
kw["proxy"] = proxy
try:
async with websockets.connect(USER_WS, **kw) as ws:
await ws.send(json.dumps({
"type": "user",
"auth": {"apiKey": creds.api_key, "secret": creds.api_secret,
"passphrase": creds.api_passphrase},
"markets": markets,
}))
try:
raw = await asyncio.wait_for(ws.recv(), timeout=4)
low = raw.lower() if isinstance(raw, str) else ""
if "auth" in low and any(w in low for w in ("fail", "error", "invalid", "unauthor")):
return False, "auth rejected by server"
return True, "connected, receiving events"
except TimeoutError:
# no message but the socket stayed open => auth accepted, just idle
return True, "connected (idle — no events yet)"
except websockets.ConnectionClosed:
return False, "connection closed (auth likely rejected)"
except Exception as e: # noqa: BLE001
return False, str(e)[:80]
async def _top_political_token(cfg: Config) -> str | None:
try:
async with httpx.AsyncClient(timeout=15) as c:
r = await c.get(f"{cfg.wallet.gamma_host}/markets",
params={"limit": 1, "closed": "false", "tag_id": 2,
"order": "volume24hr", "ascending": "false"})
toks = json.loads(r.json()[0]["clobTokenIds"])
return str(toks[0])
except (httpx.HTTPError, KeyError, IndexError, ValueError):
return None
def _extract_balance(ba: dict[str, Any]) -> float | None:
if not isinstance(ba, dict):
return None
for k in ("balance", "collateral", "amount"):
if k in ba:
try:
v = float(ba[k])
return v / 1e6 if v > 1e6 else v
except (ValueError, TypeError):
return None
return None
src/polymaker/domain.py
+185
View File
@@ -0,0 +1,185 @@
"""Core domain types shared across polymaker.
These are plain, immutable-ish dataclasses and enums with no I/O. Everything the
strategy, execution, and state layers speak is defined here so the boundaries
between components are typed rather than dict-shaped (the v1 failure mode).
"""
from __future__ import annotations
import time
from dataclasses import dataclass, field
from enum import Enum
class Side(str, Enum):
"""Order side. Values match the CLOB API's string form."""
BUY = "BUY"
SELL = "SELL"
@property
def opposite(self) -> Side:
return Side.SELL if self is Side.BUY else Side.BUY
class Regime(str, Enum):
"""Per-market quoting regime (see docs/scoping/04-strategy.md §5)."""
QUIET = "QUIET" # farming posture: in-band, layered, full size
TRENDING = "TRENDING" # persistent one-sided flow: lean + widen + half size
EVENT = "EVENT" # sweep/jump detected: pull quotes, cool off
REDUCE_ONLY = "REDUCE_ONLY" # inventory cap / end-date: exit quotes only
HALTED = "HALTED" # stale data / resolved / kill switch: cancel all
class OrderState(str, Enum):
"""Lifecycle of one of our orders."""
DRAFT = "DRAFT"
POSTED = "POSTED"
LIVE = "LIVE"
PARTIALLY_FILLED = "PARTIALLY_FILLED"
CANCELED = "CANCELED"
REJECTED = "REJECTED"
DONE = "DONE"
class TradeState(str, Enum):
"""Lifecycle of an on-chain match, mirroring the user-WS status ladder."""
MATCHED = "MATCHED"
MINED = "MINED"
CONFIRMED = "CONFIRMED"
RETRYING = "RETRYING"
FAILED = "FAILED"
# ── Market metadata ────────────────────────────────────────────────────────
@dataclass(frozen=True, slots=True)
class TokenMeta:
token_id: str
outcome: str # e.g. "Yes" / "No" / candidate name
@dataclass(frozen=True, slots=True)
class MarketMeta:
"""Static-ish metadata for a tradable market, sourced from Gamma/CLOB."""
condition_id: str
question: str
slug: str
tokens: tuple[TokenMeta, TokenMeta]
tick_size: float
neg_risk: bool
min_order_size: float # exchange minimum order size (shares)
# liquidity-rewards params
rewards_min_size: float
rewards_max_spread: float # in cents (e.g. 3.0 == 3c band)
rewards_daily_rate: float
# fees
maker_fee_bps: int
taker_fee_bps: int
fees_enabled: bool
# lifecycle / grouping
end_date_iso: str | None
event_id: str | None # neg-risk event group; siblings share this
# fraction of taker fees rebated to makers (V2 maker rebates)
rebate_rate: float = 0.0
# market-data references (may be stale; not authoritative for quoting)
best_bid: float = 0.0
best_ask: float = 0.0
liquidity_num: float = 0.0
volume_num: float = 0.0
@property
def yes(self) -> TokenMeta:
return self.tokens[0]
@property
def no(self) -> TokenMeta:
return self.tokens[1]
def other_token(self, token_id: str) -> str:
a, b = self.tokens
return b.token_id if token_id == a.token_id else a.token_id
@property
def price_decimals(self) -> int:
"""Number of decimal places implied by the tick size."""
s = f"{self.tick_size:f}".rstrip("0")
return len(s.split(".")[1]) if "." in s else 0
# ── Live trading state ─────────────────────────────────────────────────────
@dataclass(slots=True)
class Position:
token_id: str
size: float = 0.0 # signed shares held (long only in practice; >= 0)
avg_price: float = 0.0
@property
def is_flat(self) -> bool:
return self.size <= 0.0
@dataclass(slots=True)
class OpenOrder:
"""One of our resting orders as we currently believe it exists."""
order_id: str
token_id: str
side: Side
price: float
size: float # remaining (original - matched)
state: OrderState = OrderState.LIVE
created_ts: float = field(default_factory=time.time)
@property
def notional(self) -> float:
return self.price * self.size
@dataclass(frozen=True, slots=True)
class Fill:
token_id: str
side: Side
price: float
size: float
trade_id: str
ts: float = field(default_factory=time.time)
is_maker: bool = True
# ── Strategy output ────────────────────────────────────────────────────────
@dataclass(frozen=True, slots=True)
class Quote:
"""One intended resting order the strategy wants live."""
token_id: str
side: Side
price: float
size: float
def key(self, price_decimals: int) -> tuple[str, Side, float]:
"""Identity used to match against live orders (side + rounded price)."""
return (self.token_id, self.side, round(self.price, price_decimals))
@dataclass(frozen=True, slots=True)
class TargetQuotes:
"""The full desired resting-order set for a market at a point in time."""
condition_id: str
regime: Regime
quotes: tuple[Quote, ...] = ()
@property
def is_empty(self) -> bool:
return len(self.quotes) == 0
src/polymaker/engine.py
+367
View File
@@ -0,0 +1,367 @@
"""Engine: wires every component into a single async event loop.
Data flow per market:
market WS -> OrderBook -> (wake) -> Quoter task -> strategy (pure) -> reconcile
-> ExecutionGateway ; user WS -> StateStore ; periodic REST reconcile + heartbeat.
One lightweight quoter task per market, woken by book/fill events and debounced.
The strategy layer is pure; the engine owns all the state and I/O around it.
"""
from __future__ import annotations
import asyncio
import contextlib
import time
from datetime import datetime
from typing import Any
from polymaker.catalog.gamma import GammaClient, fetch_reward_rates, parse_market
from polymaker.catalog.store import CatalogStore
from polymaker.config import Config, StrategyProfile
from polymaker.domain import Fill, MarketMeta
from polymaker.execution.gateway import ExecutionGateway
from polymaker.execution.reconciler import reconcile
from polymaker.journal import Journal
from polymaker.logging import get_logger
from polymaker.marketdata.parse import TradePrint
from polymaker.marketdata.service import MarketDataService
from polymaker.merge import Merger
from polymaker.risk.manager import RiskManager
from polymaker.state.store import StateStore
from polymaker.state.tracker import UserEventProcessor
from polymaker.strategy.estimators import (
FlowEstimator,
MarketEstimators,
MarkoutTracker,
VolEstimator,
)
from polymaker.strategy.quoting import QuoteInputs, compute_fair_value, construct_quotes
from polymaker.strategy.regime import RegimeInputs, RegimeMachine
from polymaker.userstream.client import UserStream
log = get_logger("engine")
class Engine:
def __init__(self, cfg: Config, *, paper: bool = False) -> None:
self.cfg = cfg
self.paper = paper
self._running = False
self.journal = Journal(cfg.paths.journal_dir, enabled=cfg.engine.journal,
day="paper" if paper else "live")
self.state = StateStore(cfg.paths.db)
self.catalog = CatalogStore(cfg.paths.db)
self.gateway = ExecutionGateway(cfg, self.journal, paper=paper)
self.risk = RiskManager(cfg.risk, self.state)
self.merger = Merger(cfg)
self.md = MarketDataService(on_dirty=self._on_dirty, on_trade=self._on_trade,
journal=self.journal, proxy=cfg.proxy)
self.user_proc = UserEventProcessor(self.state, on_change=self._wake_cid,
on_fill=self._on_fill)
self.user: UserStream | None = None
# per-market state
self.metas: dict[str, MarketMeta] = {}
self.profiles: dict[str, StrategyProfile] = {}
self.est: dict[str, MarketEstimators] = {}
self.regime_m: dict[str, RegimeMachine] = {}
self._dirty: dict[str, asyncio.Event] = {}
self._sweep: dict[str, bool] = {}
self._merging: set[str] = set()
self._token_cid: dict[str, str] = {}
self._tasks: list[asyncio.Task[Any]] = []
# ── lifecycle ───────────────────────────────────────────────────────
async def start(self) -> None:
self._running = True
await self.gateway.connect()
await self._resolve_markets()
if not self.metas:
log.warning("no_markets_selected", hint="add markets to config/markets.toml, run `polymaker scan`")
await self._startup_reconcile()
# subscribe feeds
self.md.set_markets([(cid, [m.yes.token_id, m.no.token_id]) for cid, m in self.metas.items()])
self.user = UserStream(
self.gateway.creds, self.gateway.address, self.user_proc,
other_token=self._other_token, condition_of_token=self._cid_of_token,
journal=self.journal, proxy=self.cfg.proxy,
)
self.user.set_markets(list(self.metas))
# launch tasks
self._tasks.append(asyncio.create_task(self.md.run(), name="market_ws"))
if not self.paper:
self._tasks.append(asyncio.create_task(self.user.run(), name="user_ws"))
self._tasks.append(asyncio.create_task(self._heartbeat_loop(), name="heartbeat"))
self._tasks.append(asyncio.create_task(self._reconcile_loop(), name="reconcile"))
for cid in self.metas:
self._tasks.append(asyncio.create_task(self._quoter(cid), name=f"quote:{cid[:8]}"))
self.risk.reset_day()
log.info("engine_started", markets=len(self.metas), paper=self.paper)
async def run_forever(self) -> None:
await self.start()
with contextlib.suppress(asyncio.CancelledError):
await asyncio.gather(*self._tasks)
async def shutdown(self) -> None:
self._running = False
log.info("engine_shutdown")
self.md.stop()
if self.user:
self.user.stop()
for t in self._tasks:
t.cancel()
with contextlib.suppress(Exception):
await self.gateway.cancel_all()
self.journal.close()
self.state.close()
self.catalog.close()
# ── market resolution ───────────────────────────────────────────────
async def _resolve_markets(self) -> None:
reward_rates: dict[str, float] | None = None
async with GammaClient(self.cfg.wallet.gamma_host) as gamma:
for entry in self.cfg.enabled_markets:
meta = self.catalog.get_by_slug(entry.slug) if entry.slug else None
if meta is None and entry.condition_id:
meta = self.catalog.get(entry.condition_id)
if meta is None: # fall back to a live Gamma fetch
if reward_rates is None:
reward_rates = await fetch_reward_rates(self.cfg.wallet.clob_host)
meta = await self._fetch_meta(gamma, entry.slug, entry.condition_id, reward_rates)
if meta is None:
log.warning("market_unresolved", ref=entry.ref)
continue
self.metas[meta.condition_id] = meta
self.profiles[meta.condition_id] = self.cfg.profile_for(entry)
self.est[meta.condition_id] = self._make_estimators(self.profiles[meta.condition_id])
self.regime_m[meta.condition_id] = RegimeMachine()
self._dirty[meta.condition_id] = asyncio.Event()
for tok in (meta.yes.token_id, meta.no.token_id):
self._token_cid[tok] = meta.condition_id
async def _fetch_meta(
self, gamma: GammaClient, slug: str | None, condition_id: str | None,
reward_rates: dict[str, float],
) -> MarketMeta | None:
tag_id = self.catalog.cached_tag("politics")
async for raw in gamma.iter_markets(tag_id=tag_id, max_pages=25):
if (slug and raw.get("slug") == slug) or (condition_id and raw.get("conditionId") == condition_id):
m = parse_market(raw, reward_rates)
if m:
self.catalog.upsert_market(m)
return m
return None
@staticmethod
def _make_estimators(p: StrategyProfile) -> MarketEstimators:
return MarketEstimators(
vol=VolEstimator(p.vol_short_halflife_s, p.vol_long_halflife_s),
flow=FlowEstimator(p.flow_ewma_halflife_s),
markout=MarkoutTracker(),
)
async def _startup_reconcile(self) -> None:
with contextlib.suppress(Exception):
await self.gateway.cancel_all() # clean slate; heartbeat covers crashes
positions = await self.gateway.positions()
if positions:
self.state.reconcile_positions(positions)
log.info("startup_positions", n=len(positions))
# ── callbacks ───────────────────────────────────────────────────────
def _on_dirty(self, condition_id: str, token_id: str) -> None:
ev = self._dirty.get(condition_id)
if ev is not None:
ev.set()
def _wake_cid(self, condition_id: str) -> None:
ev = self._dirty.get(condition_id)
if ev is not None:
ev.set()
def _on_trade(self, tp: TradePrint) -> None:
cid = self._token_cid.get(tp.asset_id)
if cid is None:
return
self.est[cid].flow.update(tp.aggressor, tp.size, tp.ts)
# crude sweep flag: a single print larger than 3x base size
base = self.profiles[cid].base_size_usdc / max(tp.price, 0.01)
if tp.size >= 3 * base:
self._sweep[cid] = True
def _on_fill(self, fill: Fill) -> None:
self.risk.note_fill(fill)
cid = self._token_cid.get(fill.token_id)
if cid is None:
return
est = self.est[cid]
fv = est.last_fv if est.last_fv is not None else fill.price
token_fv = fv if fill.token_id == self.metas[cid].yes.token_id else (1.0 - fv)
est.markout.record_fill(fill.side, token_fv, fill.ts)
# ── quoter ──────────────────────────────────────────────────────────
async def _quoter(self, cid: str) -> None:
debounce = self.cfg.engine.debounce_ms / 1000.0
ev = self._dirty[cid]
while self._running:
try:
await ev.wait()
await asyncio.sleep(debounce) # coalesce a burst of book updates
ev.clear()
await self._recompute(cid)
except asyncio.CancelledError:
break
except Exception as exc: # noqa: BLE001
log.error("quoter_error", cid=cid[:8], err=str(exc))
await asyncio.sleep(0.5)
async def _recompute(self, cid: str) -> None:
meta = self.metas[cid]
p = self.profiles[cid]
yes_book = self.md.book(meta.yes.token_id)
no_book = self.md.book(meta.no.token_id)
if yes_book is None or yes_book.is_empty:
return
now = time.time()
micro = yes_book.microprice(p.micro_levels)
if micro is None:
return
est = self.est[cid]
est.flow.decay_to(now)
fv = compute_fair_value(micro, est.flow.z, meta.tick_size)
prev_fv = est.last_fv
est.on_fair_value(fv, now)
self.risk.update_mark(meta.yes.token_id, fv)
self.risk.update_mark(meta.no.token_id, 1.0 - fv)
pos_yes = self.state.position(meta.yes.token_id)
pos_no = self.state.position(meta.no.token_id)
q_max = p.q_max_usdc
inv_util = abs(pos_yes.size - pos_no.size) * fv / q_max if q_max > 0 else 0.0
hours_to_end = _hours_to_end(meta.end_date_iso, now)
ws_stale = (now - self.md.last_update_ts(meta.yes.token_id)) > self.cfg.risk.ws_stale_halt_s
rd = self.risk.evaluate(meta, ws_stale=ws_stale,
event_group_cost=self._event_group_cost(meta))
regime = self.regime_m[cid].decide(
RegimeInputs(
now=now, tick=meta.tick_size, fv=fv, prev_fv=prev_fv,
vol_ratio=est.vol.ratio, flow_z=est.flow.z, inventory_util=inv_util,
hours_to_end=hours_to_end, sweep_flagged=self._sweep.pop(cid, False),
ws_stale=ws_stale, risk_halt=rd.halt, risk_reduce_only=rd.reduce_only,
),
p,
)
tq = construct_quotes(QuoteInputs(
meta=meta, regime=regime, fv=fv, vol_short=est.vol.short,
toxicity=est.markout.toxicity, yes_view=yes_book.view(),
no_view=(no_book.view() if no_book else _empty_view()),
pos_yes=pos_yes, pos_no=pos_no, profile=p, now=now,
risk_size_scale=rd.size_scale,
))
live = self.state.orders_for(meta.yes.token_id) + self.state.orders_for(meta.no.token_id)
plan = reconcile(tq, live, tick=meta.tick_size,
reprice_ticks=p.reprice_ticks, resize_frac=p.resize_frac)
if plan.is_noop:
self._maybe_merge(cid, meta, p, pos_yes.size, pos_no.size)
return
if plan.to_cancel:
await self.gateway.cancel(plan.to_cancel)
for oid in plan.to_cancel:
self.state.remove_order(oid)
if plan.to_place:
placed = await self.gateway.place(plan.to_place, meta)
self.risk.note_order_result(bool(placed) or not plan.to_place)
for o in placed:
self.state.upsert_order(o)
log.info("requote", cid=cid[:8], regime=regime.value, fv=round(fv, 4),
place=len(plan.to_place), cancel=len(plan.to_cancel),
pos_yes=round(pos_yes.size, 1), pos_no=round(pos_no.size, 1))
self._maybe_merge(cid, meta, p, pos_yes.size, pos_no.size)
def _maybe_merge(self, cid: str, meta: MarketMeta, p: StrategyProfile,
yes_size: float, no_size: float) -> None:
amount = min(yes_size, no_size)
if amount < p.merge_min_size or cid in self._merging or self.paper:
return
self._merging.add(cid)
self._tasks.append(asyncio.create_task(self._merge_task(cid, meta, amount)))
async def _merge_task(self, cid: str, meta: MarketMeta, amount: float) -> None:
try:
raw = int(amount * 1e6)
await asyncio.to_thread(self.merger.merge, meta.condition_id, raw, meta.neg_risk)
finally:
self._merging.discard(cid)
# ── background loops ────────────────────────────────────────────────
async def _heartbeat_loop(self) -> None:
if not self.cfg.engine.heartbeat:
return
while self._running:
await self.gateway.heartbeat()
await asyncio.sleep(self.cfg.engine.heartbeat_interval_s)
async def _reconcile_loop(self) -> None:
while self._running:
await asyncio.sleep(self.cfg.engine.reconcile_interval_s)
try:
positions = await self.gateway.positions()
if positions:
self.state.reconcile_positions(positions)
live = await self.gateway.open_orders()
if live or not self.paper:
by_token: dict[str, list[Any]] = {}
for o in live:
by_token.setdefault(o.token_id, []).append(o)
for tok, orders in by_token.items():
if self.state.inflight(tok) == 0:
self.state.replace_open_orders(tok, orders)
except Exception as exc: # noqa: BLE001
log.warning("reconcile_error", err=str(exc))
# ── helpers ─────────────────────────────────────────────────────────
def _other_token(self, token_id: str) -> str | None:
cid = self._token_cid.get(token_id)
return self.metas[cid].other_token(token_id) if cid else None
def _cid_of_token(self, token_id: str) -> str | None:
return self._token_cid.get(token_id)
def _event_group_cost(self, meta: MarketMeta) -> float:
if not meta.event_id:
return 0.0
cost = 0.0
for m in self.metas.values():
if m.event_id == meta.event_id:
for tok in (m.yes.token_id, m.no.token_id):
pos = self.state.position(tok)
cost += pos.size * pos.avg_price
return cost
def _hours_to_end(end_date_iso: str | None, now: float) -> float | None:
if not end_date_iso:
return None
try:
dt = datetime.fromisoformat(end_date_iso.replace("Z", "+00:00"))
return max(0.0, (dt.timestamp() - now) / 3600.0)
except (ValueError, TypeError):
return None
def _empty_view() -> Any:
from polymaker.marketdata.orderbook import BookView
return BookView(None, 0.0, None, 0.0, None, None, 0.0, 0.0)
poly_stats/__init__.py → src/polymaker/execution/__init__.py
View File
src/polymaker/execution/gateway.py
+287
View File
@@ -0,0 +1,287 @@
"""ExecutionGateway: the only component that sends actions to the CLOB.
Wraps the synchronous py-clob-client-v2 (which owns the hard V2 EIP-712 signing,
pUSD balance adjustment, and tick/fee caching) and offloads its blocking network
calls to a thread pool so the asyncio hot path never stalls. Every quote goes out
**post-only** (the maker-only mandate, enforced at the exchange).
A `paper=True` gateway shares the same path but fabricates order ids instead of
posting — so paper mode exercises the full pipeline.
"""
from __future__ import annotations
import asyncio
import itertools
import time
from dataclasses import asdict
from typing import Any
import httpx
from polymaker.config import Config
from polymaker.domain import MarketMeta, OpenOrder, OrderState, Quote, Side
from polymaker.execution.ratelimit import TokenBucket
from polymaker.journal import Journal
from polymaker.logging import get_logger
log = get_logger("execution.gateway")
def _tick_str(tick: float) -> str:
return f"{tick:g}"
class ExecutionGateway:
def __init__(
self,
cfg: Config,
journal: Journal | None = None,
*,
paper: bool = False,
) -> None:
self._cfg = cfg
self._paper = paper
self._journal = journal
self._client: Any = None # py_clob_client_v2.ClobClient
self._creds: Any = None
self._address: str = "" # signer EOA
self._funder: str = "" # funds/positions live here (proxy/deposit wallet)
self._data_host = cfg.wallet.data_api_host
# rate budgets: fraction of documented POST/DELETE ceilings (per second)
f = cfg.execution.rate_budget_fraction
self._order_bucket = TokenBucket(rate_per_s=200.0 * f, burst=500.0 * f)
self._cancel_bucket = TokenBucket(rate_per_s=200.0 * f, burst=500.0 * f)
self._paper_ids = itertools.count(1)
@property
def paper(self) -> bool:
return self._paper
@property
def creds(self) -> Any:
return self._creds
@property
def address(self) -> str:
"""The signing EOA address."""
return self._address
@property
def funder(self) -> str:
"""The address holding funds/positions (proxy/deposit wallet, or the EOA)."""
return self._funder or self._address
# ── lifecycle ───────────────────────────────────────────────────────
async def connect(self) -> None:
"""Build the client and derive L2 API creds (network). No-op fields in paper."""
sec = self._cfg.secrets
if self._paper and not sec.has_wallet:
# paper mode runs the full pipeline without a wallet (no orders posted)
self._address = sec.browser_address or "0xPAPER"
self._funder = sec.browser_address or self._address
log.info("gateway_connected", address=self._address[:10], paper=True)
return
if not sec.has_wallet:
raise RuntimeError("no wallet configured (set PK and BROWSER_ADDRESS in .env)")
def _build() -> tuple[Any, Any, str]:
from py_clob_client_v2.client import ClobClient
client = ClobClient(
host=self._cfg.wallet.clob_host,
chain_id=self._cfg.wallet.chain_id,
key=sec.pk,
signature_type=self._cfg.wallet.signature_type,
funder=sec.browser_address,
)
creds = client.create_or_derive_api_key()
client.set_api_creds(creds)
return client, creds, client.get_address()
self._client, self._creds, self._address = await asyncio.to_thread(_build)
# funds/positions live on the funder (proxy/deposit wallet); fall back to EOA
self._funder = sec.browser_address or self._address
log.info("gateway_connected", signer=self._address[:10], funder=self._funder[:10],
paper=self._paper)
# ── placement ───────────────────────────────────────────────────────
async def place(self, quotes: list[Quote], meta: MarketMeta) -> list[OpenOrder]:
if not quotes:
return []
await self._order_bucket.acquire(len(quotes))
ts = time.time()
self._journal_write("orders_out", [asdict(q) for q in quotes], ts)
if self._paper:
return [self._paper_order(q) for q in quotes]
def _place() -> list[OpenOrder]:
from py_clob_client_v2.clob_types import (
OrderArgsV2,
OrderType,
PartialCreateOrderOptions,
PostOrdersV2Args,
)
opts = PartialCreateOrderOptions(tick_size=_tick_str(meta.tick_size), neg_risk=meta.neg_risk)
args = []
for q in quotes:
signed = self._client.create_order(
OrderArgsV2(token_id=q.token_id, price=q.price, size=q.size, side=q.side.value),
options=opts,
)
args.append(PostOrdersV2Args(order=signed, orderType=OrderType.GTC))
resp = self._client.post_orders(args, post_only=self._cfg.execution.post_only)
return self._parse_place_response(resp, quotes)
try:
return await asyncio.to_thread(_place)
except Exception as exc: # noqa: BLE001 - surface + continue; engine handles error rate
log.error("place_failed", err=str(exc), n=len(quotes))
return []
def _paper_order(self, q: Quote) -> OpenOrder:
oid = f"paper-{next(self._paper_ids)}"
return OpenOrder(oid, q.token_id, q.side, q.price, q.size, OrderState.LIVE)
def _parse_place_response(self, resp: Any, quotes: list[Quote]) -> list[OpenOrder]:
"""Map a batch post response to OpenOrders. Tolerant of shape variants;
the user-WS order events + REST snapshot reconcile anything we miss."""
items = resp if isinstance(resp, list) else resp.get("orders", resp.get("data", []))
out: list[OpenOrder] = []
for q, item in zip(quotes, items if isinstance(items, list) else [], strict=False):
oid = _first(item, "orderID", "orderId", "order_id", "id", "hash")
if not oid:
log.warning("place_response_missing_id", item=str(item)[:120])
continue
out.append(OpenOrder(str(oid), q.token_id, q.side, q.price, q.size, OrderState.LIVE))
return out
# ── cancellation ────────────────────────────────────────────────────
async def cancel(self, order_ids: list[str]) -> None:
if not order_ids or self._paper:
return
await self._cancel_bucket.acquire(1)
def _cancel() -> None:
self._client.cancel_orders(order_ids)
try:
await asyncio.to_thread(_cancel)
except Exception as exc: # noqa: BLE001
log.error("cancel_failed", err=str(exc), n=len(order_ids))
async def cancel_asset(self, asset_id: str) -> None:
if self._paper:
return
def _cancel() -> None:
from py_clob_client_v2.clob_types import OrderMarketCancelParams
self._client.cancel_market_orders(OrderMarketCancelParams(asset_id=asset_id))
await asyncio.to_thread(_cancel)
async def cancel_all(self) -> None:
if self._paper or self._client is None:
return
await asyncio.to_thread(self._client.cancel_all)
log.info("cancel_all_sent")
# ── heartbeat (dead-man switch) ─────────────────────────────────────
async def heartbeat(self, hb_id: str = "") -> None:
if self._paper or self._client is None:
return
try:
await asyncio.to_thread(self._client.post_heartbeat, hb_id)
except Exception as exc: # noqa: BLE001
log.warning("heartbeat_failed", err=str(exc))
# ── reads ───────────────────────────────────────────────────────────
async def open_orders(self) -> list[OpenOrder]:
if self._paper or self._client is None:
return []
def _get() -> list[OpenOrder]:
raw = self._client.get_open_orders()
rows = raw if isinstance(raw, list) else raw.get("data", raw.get("orders", []))
out = []
for r in rows:
try:
side = Side(str(r["side"]).upper())
remaining = float(r.get("original_size", r.get("size", 0))) - float(
r.get("size_matched", 0)
)
out.append(
OpenOrder(
str(_first(r, "id", "orderID", "order_id")),
str(r["asset_id"]),
side,
float(r["price"]),
remaining,
OrderState.LIVE,
)
)
except (KeyError, ValueError, TypeError):
continue
return out
try:
return await asyncio.to_thread(_get)
except Exception as exc: # noqa: BLE001
log.warning("open_orders_failed", err=str(exc))
return []
async def positions(self) -> dict[str, tuple[float, float]]:
"""{token_id: (size, avg_price)} from the data API (reconcile use).
Queries the FUNDER (where positions live), not the signer EOA.
"""
user = self.funder
if not user or not user.startswith("0x") or user == "0xPAPER":
return {}
try:
async with httpx.AsyncClient(timeout=15.0) as c:
r = await c.get(f"{self._data_host}/positions", params={"user": user})
r.raise_for_status()
return {
str(p["asset"]): (float(p["size"]), float(p.get("avgPrice", 0)))
for p in r.json()
if float(p.get("size", 0)) > 0
}
except (httpx.HTTPError, KeyError, ValueError) as exc:
log.warning("positions_failed", err=str(exc))
return {}
async def balance_allowance(self) -> dict[str, Any]:
"""Collateral balance/allowance snapshot (for `doctor`)."""
if self._client is None:
return {}
def _get() -> dict[str, Any]:
from py_clob_client_v2.clob_types import AssetType, BalanceAllowanceParams
result: dict[str, Any] = self._client.get_balance_allowance(
BalanceAllowanceParams(asset_type=AssetType.COLLATERAL)
)
return result
try:
return await asyncio.to_thread(_get)
except Exception as exc: # noqa: BLE001
log.warning("balance_allowance_failed", err=str(exc))
return {}
def _journal_write(self, kind: str, payload: Any, ts: float) -> None:
if self._journal is not None:
self._journal.write(kind, payload, ts)
def _first(d: Any, *keys: str) -> Any:
if not isinstance(d, dict):
return None
for k in keys:
if k in d and d[k]:
return d[k]
return None
src/polymaker/execution/ratelimit.py
+42
View File
@@ -0,0 +1,42 @@
"""Async token-bucket rate budgeter.
The CLOB API throttles (queues) excess requests rather than 429-ing, so the real
risk is silent latency injection when the market is moving. We self-limit to a
fraction of the documented ceilings and expose pressure as a signal so the
engine can shed low-edge reprices first.
"""
from __future__ import annotations
import asyncio
import time
class TokenBucket:
def __init__(self, rate_per_s: float, burst: float | None = None) -> None:
self.rate = max(rate_per_s, 0.001)
self.capacity = burst if burst is not None else max(1.0, rate_per_s)
self._tokens = self.capacity
self._last = time.monotonic()
self._lock = asyncio.Lock()
def _refill(self) -> None:
now = time.monotonic()
self._tokens = min(self.capacity, self._tokens + (now - self._last) * self.rate)
self._last = now
async def acquire(self, n: float = 1.0) -> None:
async with self._lock:
while True:
self._refill()
if self._tokens >= n:
self._tokens -= n
return
deficit = n - self._tokens
await asyncio.sleep(deficit / self.rate)
@property
def pressure(self) -> float:
"""0 = plenty of budget, 1 = empty (callers about to wait)."""
self._refill()
return 1.0 - min(1.0, self._tokens / self.capacity)
src/polymaker/execution/reconciler.py
+64
View File
@@ -0,0 +1,64 @@
"""Pure reconciliation: desired TargetQuotes vs live orders -> minimal actions.
The strategy emits a target quote set; this computes the smallest cancel/place
set to reach it, applying churn tolerances so we don't burn queue position for
sub-tick or sub-threshold size changes (v1's should_cancel instinct, generalized).
No I/O — the gateway executes the returned plan.
"""
from __future__ import annotations
from collections import defaultdict
from dataclasses import dataclass, field
from polymaker.domain import OpenOrder, Quote, TargetQuotes
_EPS = 1e-9
@dataclass(frozen=True, slots=True)
class ReconcilePlan:
to_cancel: list[str] = field(default_factory=list) # order ids
to_place: list[Quote] = field(default_factory=list)
@property
def is_noop(self) -> bool:
return not self.to_cancel and not self.to_place
def reconcile(
targets: TargetQuotes,
live: list[OpenOrder],
*,
tick: float,
reprice_ticks: int,
resize_frac: float,
) -> ReconcilePlan:
"""Diff targets against live orders. Keep live orders that already satisfy a
target within tolerance; cancel the rest; place targets with no match."""
live_by_key: dict[tuple[str, str], list[OpenOrder]] = defaultdict(list)
for o in live:
live_by_key[(o.token_id, o.side.value)].append(o)
keep: set[str] = set()
to_place: list[Quote] = []
price_tol = reprice_ticks * tick + _EPS
for q in targets.quotes:
candidates = live_by_key.get((q.token_id, q.side.value), [])
match: OpenOrder | None = None
for o in candidates:
if o.order_id in keep:
continue
price_close = abs(o.price - q.price) <= price_tol
size_close = q.size <= 0 or abs(o.size - q.size) <= resize_frac * q.size + _EPS
if price_close and size_close:
match = o
break
if match is not None:
keep.add(match.order_id)
else:
to_place.append(q)
to_cancel = [o.order_id for o in live if o.order_id not in keep]
return ReconcilePlan(to_cancel=to_cancel, to_place=to_place)
src/polymaker/journal.py
+32
View File
@@ -0,0 +1,32 @@
"""Append-only JSONL event journal.
Captures raw WS-in and orders-out so the replay backtester (docs 04 §9) can
reconstruct books and re-run the strategy. Also the substrate for post-mortems.
Cheap: one line per event, flushed, rotated by day.
"""
from __future__ import annotations
import json
from pathlib import Path
from typing import Any
class Journal:
def __init__(self, directory: str | Path, *, enabled: bool = True, day: str = "live") -> None:
self.enabled = enabled
self._fh = None
if enabled:
d = Path(directory)
d.mkdir(parents=True, exist_ok=True)
self._fh = (d / f"{day}.jsonl").open("a", buffering=1)
def write(self, kind: str, payload: Any, ts: float) -> None:
if not self.enabled or self._fh is None:
return
self._fh.write(json.dumps({"ts": ts, "kind": kind, "data": payload}, default=str) + "\n")
def close(self) -> None:
if self._fh is not None:
self._fh.close()
self._fh = None
src/polymaker/livetest.py
+94
View File
@@ -0,0 +1,94 @@
"""Live wallet round-trip test — the Phase-2 wallet spike (docs 06).
Proves the full V2 order path against the real exchange with minimal risk:
places ONE post-only BUY well below the touch (so it rests and cannot fill),
confirms it appears in open orders, then cancels it. Post-only guarantees it
never takes; the deep price + immediate cancel means ~zero economic risk.
This is where the known py-clob-client-v2 signature-type-2 (Safe/proxy) issues
would surface — the command reports each step so failures are diagnosable.
"""
from __future__ import annotations
import asyncio
from rich.console import Console
from polymaker.config import Config
from polymaker.domain import Quote, Side
from polymaker.execution.gateway import ExecutionGateway
async def run_livetest(cfg: Config, console: Console, notional_usdc: float = 5.0) -> bool:
from polymaker.catalog.store import CatalogStore
if not cfg.secrets.has_wallet:
console.print("[red]No wallet in .env. Set PK and BROWSER_ADDRESS first.[/red]")
return False
# pick a liquid market from the catalog (or fall back to a live scan)
store = CatalogStore(cfg.paths.db)
rows = store.top(20)
store.close()
if not rows:
console.print("[yellow]Catalog empty — run `polymaker scan` first.[/yellow]")
return False
# choose a market with a mid comfortably in (0.15, 0.85) so a deep bid is valid
meta = None
for m, _sc in rows:
mid = (m.best_bid + m.best_ask) / 2 if (m.best_bid and m.best_ask) else 0.0
if 0.15 < mid < 0.85:
meta = m
break
meta = meta or rows[0][0]
console.print(f"[bold]Live round-trip test[/bold] on: {meta.question[:60]}")
gw = ExecutionGateway(cfg)
try:
await gw.connect()
console.print(f" [green]✓[/green] wallet auth — address {gw.address[:12]}…")
except Exception as e: # noqa: BLE001
console.print(f" [red]✗ wallet auth failed:[/red] {e}")
console.print(" [yellow]Auth/signature-type mismatch (docs 03 §9). If your account has a "
"'deposit address', set signature_type=3 (POLY_1271) in config.toml and use the "
"deposit address as BROWSER_ADDRESS. Errors like 'maker address not allowed, use "
"the deposit wallet flow' or 'signer must be the API key address' mean the type is "
"wrong for this wallet.[/yellow]")
return False
ba = await gw.balance_allowance()
console.print(f" balance/allowance: {ba}")
# a deep resting price: well below best bid, snapped to tick, floored at 2 ticks
tick = meta.tick_size
best_bid = meta.best_bid or 0.30
price = max(2 * tick, round((best_bid - 0.10) / tick) * tick)
size = round(max(meta.min_order_size, notional_usdc / price), 2)
console.print(f" placing post-only BUY {size} @ {price} on YES token "
f"(~${price * size:.2f}, deep — will not fill)")
placed = await gw.place([Quote(meta.yes.token_id, Side.BUY, price, size)], meta)
if not placed:
console.print(" [red]✗ order not placed (see logs for the API error)[/red]")
return False
oid = placed[0].order_id
console.print(f" [green]✓[/green] placed — order id {oid[:16]}…")
await asyncio.sleep(2.0)
live = await gw.open_orders()
found = any(o.order_id == oid for o in live)
console.print(f" [{'green' if found else 'yellow'}]{'✓' if found else '?'}[/] "
f"read back open orders: {len(live)} live, ours {'present' if found else 'not seen yet'}")
await gw.cancel([oid])
console.print(" [green]✓[/green] cancel sent")
await asyncio.sleep(1.5)
after = await gw.open_orders()
still = any(o.order_id == oid for o in after)
console.print(f" [{'green' if not still else 'red'}]{'✓' if not still else '✗'}[/] "
f"order {'cancelled' if not still else 'STILL LIVE — cancel manually!'}")
ok = bool(placed) and not still
console.print(f"\n[bold]{'ROUND-TRIP OK' if ok else 'CHECK LOGS'}[/bold]")
return ok
src/polymaker/logging.py
+66
View File
@@ -0,0 +1,66 @@
"""structlog configuration: human console in dev, JSON to file in prod."""
from __future__ import annotations
import logging
import sys
from pathlib import Path
from typing import Any
import structlog
def configure(
*,
level: str = "INFO",
json_file: Path | None = None,
console: bool = True,
) -> None:
"""Set up structlog + stdlib logging once at process start."""
shared: list[Any] = [
structlog.contextvars.merge_contextvars,
structlog.processors.add_log_level,
structlog.processors.TimeStamper(fmt="iso", utc=True),
structlog.processors.StackInfoRenderer(),
structlog.processors.format_exc_info,
]
structlog.configure(
processors=[*shared, structlog.stdlib.ProcessorFormatter.wrap_for_formatter],
logger_factory=structlog.stdlib.LoggerFactory(),
wrapper_class=structlog.stdlib.BoundLogger,
cache_logger_on_first_use=True,
)
root = logging.getLogger()
root.handlers.clear()
root.setLevel(level)
if console:
ch = logging.StreamHandler(sys.stderr)
ch.setFormatter(
structlog.stdlib.ProcessorFormatter(
processors=[
structlog.stdlib.ProcessorFormatter.remove_processors_meta,
structlog.dev.ConsoleRenderer(colors=sys.stderr.isatty()),
]
)
)
root.addHandler(ch)
if json_file is not None:
json_file.parent.mkdir(parents=True, exist_ok=True)
fh = logging.FileHandler(json_file)
fh.setFormatter(
structlog.stdlib.ProcessorFormatter(
processors=[
structlog.stdlib.ProcessorFormatter.remove_processors_meta,
structlog.processors.JSONRenderer(),
]
)
)
root.addHandler(fh)
def get_logger(name: str) -> structlog.stdlib.BoundLogger:
return structlog.get_logger(name) # type: ignore[no-any-return]
poly_utils/__init__.py → src/polymaker/marketdata/__init__.py
View File
src/polymaker/marketdata/orderbook.py
+209
View File
@@ -0,0 +1,209 @@
"""Order book maintenance and analytics for a single market.
We keep the YES-token book canonical (bids/asks as SortedDicts keyed by price).
The NO-token view is derived by the identity no_price = 1 - yes_price, with
bids/asks swapped — so we only ever maintain one book per market.
All methods are synchronous and side-effect-free reads except the explicit
apply_* mutators. Nothing here does I/O; the WS layer drives it.
"""
from __future__ import annotations
from dataclasses import dataclass
from sortedcontainers import SortedDict
from polymaker.domain import Side
@dataclass(frozen=True, slots=True)
class BookLevel:
price: float
size: float
@dataclass(frozen=True, slots=True)
class BookView:
"""A resolved best/second/depth snapshot for one outcome token."""
best_bid: float | None
best_bid_size: float
best_ask: float | None
best_ask_size: float
second_bid: float | None
second_ask: float | None
bid_depth: float # summed size within band, bid side
ask_depth: float # summed size within band, ask side
@property
def mid(self) -> float | None:
if self.best_bid is None or self.best_ask is None:
return None
return (self.best_bid + self.best_ask) / 2.0
@property
def spread(self) -> float | None:
if self.best_bid is None or self.best_ask is None:
return None
return self.best_ask - self.best_bid
@property
def imbalance(self) -> float:
"""(bid_depth - ask_depth) / total in [-1, 1]; 0 if empty."""
total = self.bid_depth + self.ask_depth
return (self.bid_depth - self.ask_depth) / total if total > 0 else 0.0
class OrderBook:
"""YES-canonical L2 book for one market."""
__slots__ = ("bids", "asks", "tick_size", "last_update_ts", "book_hash")
def __init__(self, tick_size: float = 0.001) -> None:
# price -> size. bids and asks both ascending in price.
self.bids: SortedDict[float, float] = SortedDict()
self.asks: SortedDict[float, float] = SortedDict()
self.tick_size = tick_size
self.last_update_ts: float = 0.0
self.book_hash: str | None = None
# ── mutation ────────────────────────────────────────────────────────
def apply_snapshot(
self,
bids: list[tuple[float, float]],
asks: list[tuple[float, float]],
ts: float,
book_hash: str | None = None,
) -> None:
self.bids = SortedDict({p: s for p, s in bids if s > 0})
self.asks = SortedDict({p: s for p, s in asks if s > 0})
self.last_update_ts = ts
self.book_hash = book_hash
def apply_delta(self, side: Side, price: float, size: float, ts: float) -> None:
book = self.bids if side is Side.BUY else self.asks
if size <= 0:
book.pop(price, None)
else:
book[price] = size
self.last_update_ts = ts
def set_tick_size(self, tick_size: float) -> None:
self.tick_size = tick_size
@property
def is_empty(self) -> bool:
return len(self.bids) == 0 or len(self.asks) == 0
# ── raw best (YES side) ─────────────────────────────────────────────
def best_bid(self) -> BookLevel | None:
if not self.bids:
return None
p = self.bids.peekitem(-1) # highest bid
return BookLevel(p[0], p[1])
def best_ask(self) -> BookLevel | None:
if not self.asks:
return None
p = self.asks.peekitem(0) # lowest ask
return BookLevel(p[0], p[1])
# ── analytics ───────────────────────────────────────────────────────
def microprice(self, levels: int = 3) -> float | None:
"""Depth-weighted mid over the top `levels`, pulled toward the thin side.
Uses size at the opposite side as the weight for each price (standard
microprice intuition: price is dragged toward the side with less size).
Returns None if either side is empty.
"""
bb = self.best_bid()
ba = self.best_ask()
if bb is None or ba is None:
return None
bid_sz = self._top_size(self.bids, levels, from_high=True)
ask_sz = self._top_size(self.asks, levels, from_high=False)
total = bid_sz + ask_sz
if total <= 0:
return (bb.price + ba.price) / 2.0
# weight best_ask by bid size and best_bid by ask size
return (ba.price * bid_sz + bb.price * ask_sz) / total
def best_with_min_size(
self, side: Side, min_size: float
) -> tuple[float | None, float, float | None]:
"""First level (from the touch) with size > min_size.
Returns (price, size, top_price) where top_price is the actual touch
(used to detect dust at the front). Mirrors v1's find_best_price_with_size
but without the second-best bookkeeping the new strategy doesn't need.
"""
if side is Side.BUY:
items = reversed(self.bids.items()) # high -> low
else:
items = iter(self.asks.items()) # low -> high
top_price: float | None = None
for price, size in items:
if top_price is None:
top_price = price
if size > min_size:
return price, size, top_price
return None, 0.0, top_price
def depth_within(self, side: Side, lo: float, hi: float) -> float:
"""Sum of sizes with price in [lo, hi] on the given side."""
book = self.bids if side is Side.BUY else self.asks
# SortedDict.irange gives keys in [lo, hi]
return float(sum(book[p] for p in book.irange(lo, hi)))
def view(self, band_frac: float = 0.05, min_size: float = 0.0) -> BookView:
"""Resolved YES-side view with best/second and in-band depth."""
bb = self._nth_bid(0, min_size)
ba = self._nth_ask(0, min_size)
sb = self._nth_bid(1, min_size)
sa = self._nth_ask(1, min_size)
mid = None
bid_depth = ask_depth = 0.0
if bb is not None and ba is not None:
mid = (bb.price + ba.price) / 2.0
bid_depth = self.depth_within(Side.BUY, bb.price, mid * (1 + band_frac))
ask_depth = self.depth_within(Side.SELL, mid * (1 - band_frac), ba.price)
return BookView(
best_bid=bb.price if bb else None,
best_bid_size=bb.size if bb else 0.0,
best_ask=ba.price if ba else None,
best_ask_size=ba.size if ba else 0.0,
second_bid=sb.price if sb else None,
second_ask=sa.price if sa else None,
bid_depth=bid_depth,
ask_depth=ask_depth,
)
# ── internals ───────────────────────────────────────────────────────
def _nth_bid(self, n: int, min_size: float) -> BookLevel | None:
count = 0
for price in reversed(self.bids):
if self.bids[price] > min_size:
if count == n:
return BookLevel(price, self.bids[price])
count += 1
return None
def _nth_ask(self, n: int, min_size: float) -> BookLevel | None:
count = 0
for price in self.asks:
if self.asks[price] > min_size:
if count == n:
return BookLevel(price, self.asks[price])
count += 1
return None
@staticmethod
def _top_size(book: SortedDict[float, float], levels: int, *, from_high: bool) -> float:
keys = list(reversed(book)) if from_high else list(book)
return float(sum(book[k] for k in keys[:levels]))
def to_no_price(yes_price: float) -> float:
"""Convert a YES price to the equivalent NO price."""
return 1.0 - yes_price
src/polymaker/marketdata/parse.py
+136
View File
@@ -0,0 +1,136 @@
"""Pure parsers for market-WS wire messages -> structured updates.
Kept separate from the socket so they're unit-testable against captured frames.
Verified against live frames on 2026-07-05 (docs/scoping/03-api-layer.md §5):
book: {market, asset_id, bids:[{price,size}], asks:[...], timestamp, hash, tick_size}
price_change:{market, timestamp, price_changes:[{asset_id, price, size, side, hash}]}
last_trade_price:{market, asset_id, price, size, side, timestamp, fee_rate_bps}
tick_size_change:{market, asset_id, old_tick_size, new_tick_size}
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Any
from polymaker.domain import Side
@dataclass(frozen=True, slots=True)
class BookUpdate:
asset_id: str
condition_id: str
bids: list[tuple[float, float]]
asks: list[tuple[float, float]]
ts: float
book_hash: str | None
tick_size: float | None
@dataclass(frozen=True, slots=True)
class PriceChange:
asset_id: str
condition_id: str
side: Side # BUY -> bid side, SELL -> ask side
price: float
size: float
ts: float
@dataclass(frozen=True, slots=True)
class TradePrint:
asset_id: str
condition_id: str
aggressor: Side
price: float
size: float
ts: float
@dataclass(frozen=True, slots=True)
class TickSizeChange:
asset_id: str
tick_size: float
def _ts(msg: dict[str, Any]) -> float:
raw = msg.get("timestamp")
if raw is None:
return 0.0
try:
v = float(raw)
return v / 1000.0 if v > 1e12 else v # ms -> s
except (ValueError, TypeError):
return 0.0
def _levels(items: Any) -> list[tuple[float, float]]:
out: list[tuple[float, float]] = []
for it in items or []:
try:
out.append((float(it["price"]), float(it["size"])))
except (KeyError, ValueError, TypeError):
continue
return out
def parse_book(msg: dict[str, Any]) -> BookUpdate | None:
try:
tick = msg.get("tick_size")
return BookUpdate(
asset_id=str(msg["asset_id"]),
condition_id=str(msg.get("market", "")),
bids=_levels(msg.get("bids")),
asks=_levels(msg.get("asks")),
ts=_ts(msg),
book_hash=msg.get("hash"),
tick_size=float(tick) if tick is not None else None,
)
except (KeyError, ValueError, TypeError):
return None
def parse_price_changes(msg: dict[str, Any]) -> list[PriceChange]:
out: list[PriceChange] = []
ts = _ts(msg)
cond = str(msg.get("market", ""))
for ch in msg.get("price_changes", []) or []:
try:
out.append(
PriceChange(
asset_id=str(ch["asset_id"]),
condition_id=cond,
side=Side(str(ch["side"]).upper()),
price=float(ch["price"]),
size=float(ch["size"]),
ts=ts,
)
)
except (KeyError, ValueError, TypeError):
continue
return out
def parse_last_trade(msg: dict[str, Any]) -> TradePrint | None:
try:
return TradePrint(
asset_id=str(msg["asset_id"]),
condition_id=str(msg.get("market", "")),
aggressor=Side(str(msg.get("side", "BUY")).upper()),
price=float(msg["price"]),
size=float(msg["size"]),
ts=_ts(msg),
)
except (KeyError, ValueError, TypeError):
return None
def parse_tick_size_change(msg: dict[str, Any]) -> TickSizeChange | None:
try:
tick = msg.get("new_tick_size", msg.get("tick_size"))
if tick is None:
return None
return TickSizeChange(asset_id=str(msg["asset_id"]), tick_size=float(tick))
except (KeyError, ValueError, TypeError):
return None
src/polymaker/marketdata/service.py
+185
View File
@@ -0,0 +1,185 @@
"""MarketDataService: owns the market WS, maintains a book per token.
Subscribes to every YES+NO token of the markets we quote and routes each frame
to that token's OrderBook. We do NOT set `custom_feature_enabled` (verified to
broaden the feed beyond our assets); resolution is detected via catalog flags.
On every book mutation it wakes the owning market's quoter via `on_dirty`, and
feeds trade prints to `on_trade` for the flow estimator. Reconnects re-snapshot
automatically because the server sends a fresh `book` on (re)subscribe.
"""
from __future__ import annotations
import asyncio
import json
from collections.abc import Callable
from typing import Any
import websockets
from polymaker.journal import Journal
from polymaker.logging import get_logger
from polymaker.marketdata.orderbook import BookView, OrderBook
from polymaker.marketdata.parse import (
TradePrint,
parse_book,
parse_last_trade,
parse_price_changes,
parse_tick_size_change,
)
log = get_logger("marketdata.service")
DirtyCb = Callable[[str, str], None] # (condition_id, token_id)
TradeCb = Callable[[TradePrint], None]
class MarketDataService:
def __init__(
self,
url: str = "wss://ws-subscriptions-clob.polymarket.com/ws/market",
*,
on_dirty: DirtyCb | None = None,
on_trade: TradeCb | None = None,
journal: Journal | None = None,
proxy: str | None = None,
) -> None:
self._url = url
self._on_dirty = on_dirty or (lambda _c, _t: None)
self._on_trade = on_trade or (lambda _tp: None)
self._journal = journal
self._proxy = proxy
self.books: dict[str, OrderBook] = {}
self._token_condition: dict[str, str] = {}
self._subs: list[str] = []
self._ws: Any = None
self._stop = asyncio.Event()
# ── subscription management ─────────────────────────────────────────
def set_markets(self, markets: list[tuple[str, list[str]]]) -> None:
"""markets = [(condition_id, [token_ids...])]. Rebuilds the desired set."""
subs: list[str] = []
for cond, tokens in markets:
for tok in tokens:
self._token_condition[tok] = cond
self.books.setdefault(tok, OrderBook())
subs.append(tok)
self._subs = subs
def view(self, token_id: str) -> BookView:
book = self.books.get(token_id)
return book.view() if book else _empty_view()
def book(self, token_id: str) -> OrderBook | None:
return self.books.get(token_id)
def last_update_ts(self, token_id: str) -> float:
b = self.books.get(token_id)
return b.last_update_ts if b else 0.0
# ── run loop ────────────────────────────────────────────────────────
async def run(self) -> None:
backoff = 1.0
while not self._stop.is_set():
try:
await self._connect_and_listen()
backoff = 1.0
except (websockets.ConnectionClosed, OSError) as exc:
log.warning("market_ws_dropped", err=str(exc), backoff=backoff)
except Exception as exc: # noqa: BLE001
log.error("market_ws_error", err=str(exc))
if self._stop.is_set():
break
await asyncio.sleep(backoff)
backoff = min(backoff * 2, 30.0)
async def _connect_and_listen(self) -> None:
if not self._subs:
await asyncio.sleep(1.0)
return
kwargs: dict[str, Any] = {"ping_interval": 5, "ping_timeout": None}
if self._proxy:
kwargs["proxy"] = self._proxy
async with websockets.connect(self._url, **kwargs) as ws:
self._ws = ws
await ws.send(json.dumps({"assets_ids": self._subs, "type": "market"}))
log.info("market_ws_subscribed", n=len(self._subs))
async for raw in ws:
self._handle(raw)
def stop(self) -> None:
self._stop.set()
# ── message handling ────────────────────────────────────────────────
def _handle(self, raw: str | bytes) -> None:
try:
data = json.loads(raw)
except (json.JSONDecodeError, TypeError):
return
for msg in data if isinstance(data, list) else [data]:
if not isinstance(msg, dict):
continue
self._dispatch(msg)
def _dispatch(self, msg: dict[str, Any]) -> None:
et = msg.get("event_type")
if et == "book":
self._on_book(msg)
elif et == "price_change":
self._on_price_change(msg)
elif et == "last_trade_price":
self._on_last_trade(msg)
elif et == "tick_size_change":
self._on_tick_change(msg)
def _on_book(self, msg: dict[str, Any]) -> None:
upd = parse_book(msg)
if upd is None or upd.asset_id not in self.books:
return
book = self.books[upd.asset_id]
if upd.tick_size:
book.set_tick_size(upd.tick_size)
book.apply_snapshot(upd.bids, upd.asks, upd.ts, upd.book_hash)
self._journal_write("book", msg, upd.ts)
self._wake(upd.asset_id)
def _on_price_change(self, msg: dict[str, Any]) -> None:
changes = parse_price_changes(msg)
touched: set[str] = set()
for ch in changes:
book = self.books.get(ch.asset_id)
if book is None:
continue
book.apply_delta(ch.side, ch.price, ch.size, ch.ts)
touched.add(ch.asset_id)
if changes:
self._journal_write("price_change", msg, changes[0].ts)
for tok in touched:
self._wake(tok)
def _on_last_trade(self, msg: dict[str, Any]) -> None:
tp = parse_last_trade(msg)
if tp is None or tp.asset_id not in self.books:
return
self._journal_write("last_trade_price", msg, tp.ts)
self._on_trade(tp)
def _on_tick_change(self, msg: dict[str, Any]) -> None:
tc = parse_tick_size_change(msg)
if tc and tc.asset_id in self.books:
self.books[tc.asset_id].set_tick_size(tc.tick_size)
log.info("tick_size_change", token=tc.asset_id[:12], tick=tc.tick_size)
def _wake(self, token_id: str) -> None:
cond = self._token_condition.get(token_id)
if cond:
self._on_dirty(cond, token_id)
def _journal_write(self, kind: str, payload: dict[str, Any], ts: float) -> None:
if self._journal is not None:
self._journal.write(kind, payload, ts)
def _empty_view() -> BookView:
return BookView(None, 0.0, None, 0.0, None, None, 0.0, 0.0)
src/polymaker/merge.py
+142
View File
@@ -0,0 +1,142 @@
"""Native Python position merger (replaces the Node.js poly_merger subprocess).
When we hold both YES and NO in the same market, merging the pair returns
collateral (1 USDC/pUSD per pair) — a maker-only exit with zero market impact.
Two execution paths:
* EOA wallet (signature_type=0): direct contract call, fully implemented here.
* Proxy/Safe wallet (signature_type 1/2): the merge tx must be routed through
the Safe. That path is gated on the Phase-2 wallet spike (docs 03 §6) — until
then merging is skipped (logged), and inventory is exited via limit sells
instead. The bot is fully functional without it; merging just frees capital
sooner.
The V2/pUSD collateral question (does the CTF collateral resolve to pUSD post-
migration?) is also spike-gated; addresses are config-driven, not baked in.
"""
from __future__ import annotations
from typing import Any
from polymaker.config import Config
from polymaker.logging import get_logger
log = get_logger("merge")
# Polygon mainnet contracts (pre-V2 defaults; confirm collateral in the spike).
CONDITIONAL_TOKENS = "0x4D97DCd97eC945f40cF65F87097ACe5EA0476045"
NEG_RISK_ADAPTER = "0xd91E80cF2E7be2e162c6513ceD06f1dD0dA35296"
USDC_COLLATERAL = "0x2791Bca1f2de4661ED88A30C99A7a9449Aa84174"
_CTF_ABI = [
{
"name": "mergePositions",
"type": "function",
"stateMutability": "nonpayable",
"inputs": [
{"name": "collateralToken", "type": "address"},
{"name": "parentCollectionId", "type": "bytes32"},
{"name": "conditionId", "type": "bytes32"},
{"name": "partition", "type": "uint256[]"},
{"name": "amount", "type": "uint256"},
],
"outputs": [],
}
]
_NEG_RISK_ABI = [
{
"name": "mergePositions",
"type": "function",
"stateMutability": "nonpayable",
"inputs": [
{"name": "conditionId", "type": "bytes32"},
{"name": "amount", "type": "uint256"},
],
"outputs": [],
}
]
class Merger:
def __init__(self, cfg: Config) -> None:
self._cfg = cfg
self._w3: Any = None
self._account: Any = None
def _ensure_web3(self) -> None:
if self._w3 is not None:
return
from eth_account import Account
from web3 import Web3
from web3.middleware import ExtraDataToPOAMiddleware
rpc = self._cfg.secrets.polygon_rpc or self._cfg.wallet.polygon_rpc
w3 = Web3(Web3.HTTPProvider(rpc))
w3.middleware_onion.inject(ExtraDataToPOAMiddleware, layer=0)
self._w3 = w3
self._account = Account.from_key(self._cfg.secrets.pk)
@property
def can_merge(self) -> bool:
"""EOA can merge directly today; Safe/proxy is spike-gated."""
return self._cfg.wallet.signature_type == 0
def merge(self, condition_id: str, amount_raw: int, neg_risk: bool) -> str | None:
"""Merge `amount_raw` (6-dec) YES+NO pairs. Returns tx hash or None."""
if amount_raw <= 0:
return None
if not self.can_merge:
log.info(
"merge_skipped_safe_wallet",
condition=condition_id[:12],
amount=amount_raw,
note="Safe merge is spike-gated; inventory exits via limit sells",
)
return None
try:
return self._merge_eoa(condition_id, amount_raw, neg_risk)
except Exception as exc: # noqa: BLE001
log.error("merge_failed", condition=condition_id[:12], err=str(exc))
return None
def _merge_eoa(self, condition_id: str, amount_raw: int, neg_risk: bool) -> str:
self._ensure_web3()
w3 = self._w3
addr = self._account.address
cond = _to_bytes32(condition_id)
if neg_risk:
c = w3.eth.contract(address=w3.to_checksum_address(NEG_RISK_ADAPTER), abi=_NEG_RISK_ABI)
fn = c.functions.mergePositions(cond, amount_raw)
else:
c = w3.eth.contract(address=w3.to_checksum_address(CONDITIONAL_TOKENS), abi=_CTF_ABI)
fn = c.functions.mergePositions(
w3.to_checksum_address(USDC_COLLATERAL),
b"\x00" * 32, # parent collection id (top-level market)
cond,
[1, 2], # partition: the two outcome slots
amount_raw,
)
tx = fn.build_transaction(
{
"from": addr,
"nonce": w3.eth.get_transaction_count(addr),
"chainId": self._cfg.wallet.chain_id,
"gas": 300_000,
"maxFeePerGas": w3.eth.gas_price * 2,
"maxPriorityFeePerGas": w3.to_wei(30, "gwei"),
}
)
signed = self._account.sign_transaction(tx)
tx_hash = w3.eth.send_raw_transaction(signed.raw_transaction)
receipt = w3.eth.wait_for_transaction_receipt(tx_hash, timeout=120)
h = str(receipt["transactionHash"].hex())
log.info("merge_sent", condition=condition_id[:12], amount=amount_raw, tx=h[:14])
return h
def _to_bytes32(hex_str: str) -> bytes:
s = hex_str[2:] if hex_str.startswith("0x") else hex_str
return bytes.fromhex(s.rjust(64, "0"))
src/polymaker/py.typed
View File
src/polymaker/risk/__init__.py
View File
src/polymaker/risk/manager.py
+146
View File
@@ -0,0 +1,146 @@
"""RiskManager: pre-trade gates and circuit breakers (docs 04 §6, 02).
Consulted by the engine before every quote set. Returns a per-market decision
(size scale / reduce-only / halt) and owns the global kill switches. Position
and order data come from the StateStore; fair-value marks are pushed in by the
engine so PnL is always current.
"""
from __future__ import annotations
from dataclasses import dataclass
from polymaker.config import RiskConfig
from polymaker.domain import Fill, MarketMeta, Side
from polymaker.logging import get_logger
from polymaker.state.store import StateStore
log = get_logger("risk.manager")
@dataclass(frozen=True, slots=True)
class RiskDecision:
halt: bool # HALTED regime for this market
reduce_only: bool # REDUCE_ONLY regime for this market
size_scale: float # multiply quote sizes by this [0,1]
reason: str = ""
class RiskManager:
def __init__(self, cfg: RiskConfig, store: StateStore) -> None:
self._cfg = cfg
self._store = store
self._marks: dict[str, float] = {} # token_id -> fair value
self._net_cash = 0.0 # cumulative signed cash from fills (+sell, -buy)
self._day_start_equity = 0.0
self._killed = False
self._order_attempts = 0
self._order_errors = 0
# ── PnL bookkeeping ─────────────────────────────────────────────────
def note_fill(self, fill: Fill) -> None:
self._net_cash += (fill.price * fill.size) * (1 if fill.side is Side.SELL else -1)
def update_mark(self, token_id: str, fv: float) -> None:
self._marks[token_id] = fv
def _inventory_value(self) -> float:
total = 0.0
for tok, pos in self._store.positions.items():
if pos.size > 0:
total += pos.size * self._marks.get(tok, pos.avg_price)
return total
@property
def equity(self) -> float:
return self._net_cash + self._inventory_value()
@property
def daily_pnl(self) -> float:
return self.equity - self._day_start_equity
def reset_day(self) -> None:
self._day_start_equity = self.equity
# ── error-rate breaker ──────────────────────────────────────────────
def note_order_result(self, ok: bool) -> None:
self._order_attempts += 1
if not ok:
self._order_errors += 1
@property
def error_rate(self) -> float:
return self._order_errors / self._order_attempts if self._order_attempts >= 20 else 0.0
# ── global kill switch ──────────────────────────────────────────────
def global_halt(self) -> tuple[bool, str]:
if self._killed:
return True, "manual_kill"
if self.daily_pnl <= -self._cfg.daily_loss_kill_usdc:
return True, f"daily_loss {self.daily_pnl:.0f}"
if self.error_rate >= self._cfg.max_order_error_rate:
return True, f"error_rate {self.error_rate:.2f}"
return False, ""
def kill(self) -> None:
self._killed = True
log.critical("kill_switch_engaged")
# ── per-market evaluation ───────────────────────────────────────────
def evaluate(
self, meta: MarketMeta, *, ws_stale: bool, event_group_cost: float
) -> RiskDecision:
halted, why = self.global_halt()
if halted:
return RiskDecision(True, False, 0.0, why)
if ws_stale:
return RiskDecision(True, False, 0.0, "ws_stale")
market_notional = self._market_notional(meta)
total_exposure = self._total_exposure()
# hard caps -> reduce only
if market_notional >= self._cfg.max_market_notional_usdc:
return RiskDecision(False, True, 1.0, "market_cap")
if event_group_cost >= self._cfg.max_event_group_loss_usdc:
return RiskDecision(False, True, 1.0, "event_group_cap")
if total_exposure >= self._cfg.max_total_exposure_usdc:
return RiskDecision(False, True, 1.0, "total_exposure_cap")
# soft scaling: taper size as any cap is approached (worst-binding wins)
scale = min(
_headroom(market_notional, self._cfg.max_market_notional_usdc),
_headroom(total_exposure, self._cfg.max_total_exposure_usdc),
_headroom(event_group_cost, self._cfg.max_event_group_loss_usdc),
)
return RiskDecision(False, False, scale, "")
def _market_notional(self, meta: MarketMeta) -> float:
total = 0.0
for tok in (meta.yes.token_id, meta.no.token_id):
pos = self._store.position(tok)
total += pos.size * self._marks.get(tok, pos.avg_price or 0.5)
for o in self._store.orders_for(tok):
if o.side is Side.BUY:
total += o.notional
return total
def _total_exposure(self) -> float:
total = 0.0
for tok, pos in self._store.positions.items():
if pos.size > 0:
total += pos.size * self._marks.get(tok, pos.avg_price or 0.5)
for o in self._store.orders.values():
if o.side is Side.BUY:
total += o.notional
return total
def _headroom(current: float, cap: float) -> float:
"""1.0 well below the cap, tapering to 0 as we approach it (from 70%)."""
if cap <= 0:
return 1.0
frac = current / cap
if frac <= 0.7:
return 1.0
return max(0.0, (1.0 - frac) / 0.3)
src/polymaker/state/__init__.py
View File
src/polymaker/state/store.py
+176
View File
@@ -0,0 +1,176 @@
"""StateStore: the single owner of positions and open orders.
Replaces v1's module-level global dicts + the `performing`/`last_trade_update`
races. Three inputs, one arbitration rule (docs/scoping/02-architecture.md):
* WS fill events apply immediately (optimistic),
* REST reconciliation corrects drift ONLY for tokens with no in-flight trades,
* on-chain balances are consulted only by the merger.
In-memory + typed, mirrored to SQLite on change so a crash-restart resumes.
"""
from __future__ import annotations
import json
import sqlite3
import time
from pathlib import Path
from polymaker.domain import Fill, OpenOrder, OrderState, Position, Side
from polymaker.logging import get_logger
log = get_logger("state.store")
_SCHEMA = """
CREATE TABLE IF NOT EXISTS positions (
token_id TEXT PRIMARY KEY,
size REAL NOT NULL,
avg_price REAL NOT NULL,
updated_ts REAL NOT NULL
);
CREATE TABLE IF NOT EXISTS fills (
trade_id TEXT PRIMARY KEY,
token_id TEXT, side TEXT, price REAL, size REAL, is_maker INT, ts REAL
);
CREATE TABLE IF NOT EXISTS order_log (
order_id TEXT PRIMARY KEY,
token_id TEXT, side TEXT, price REAL, size REAL, state TEXT, ts REAL
);
"""
class StateStore:
"""Owns positions + open orders + a per-token in-flight guard."""
def __init__(self, db_path: str | Path = "state.db") -> None:
self._conn = sqlite3.connect(str(db_path))
self._conn.row_factory = sqlite3.Row
self._conn.execute("PRAGMA journal_mode=WAL")
self._conn.executescript(_SCHEMA)
self._conn.commit()
self.positions: dict[str, Position] = {}
# order_id -> OpenOrder
self.orders: dict[str, OpenOrder] = {}
# token_id -> count of in-flight (MATCHED-not-CONFIRMED) trades; guards reconcile
self._inflight: dict[str, int] = {}
self._last_fill_ts: dict[str, float] = {}
self._load()
def close(self) -> None:
self._conn.close()
# ── positions ───────────────────────────────────────────────────────
def position(self, token_id: str) -> Position:
return self.positions.get(token_id, Position(token_id))
def apply_fill(self, fill: Fill) -> None:
"""Apply a fill optimistically to inventory + avg price."""
pos = self.positions.setdefault(fill.token_id, Position(fill.token_id))
signed = fill.size if fill.side is Side.BUY else -fill.size
new_size = pos.size + signed
if fill.side is Side.BUY:
if pos.size <= 0:
pos.avg_price = fill.price
else:
pos.avg_price = (pos.avg_price * pos.size + fill.price * fill.size) / (
pos.size + fill.size
)
# selling leaves avg_price unchanged
pos.size = max(0.0, new_size)
if pos.size <= 0:
pos.avg_price = 0.0
self._last_fill_ts[fill.token_id] = fill.ts
self._persist_position(pos)
self._record_fill(fill)
log.info("fill", token=fill.token_id[:12], side=fill.side.value,
price=fill.price, size=fill.size, pos=round(pos.size, 2))
def set_position(self, token_id: str, size: float, avg_price: float) -> None:
pos = Position(token_id, max(0.0, size), avg_price if size > 0 else 0.0)
self.positions[token_id] = pos
self._persist_position(pos)
def reconcile_positions(self, api_positions: dict[str, tuple[float, float]]) -> None:
"""Overwrite sizes from REST, skipping tokens with in-flight trades or
a very recent fill (the optimistic value is more current there)."""
now = time.time()
for token_id, (size, avg) in api_positions.items():
if self._inflight.get(token_id, 0) > 0:
continue
if now - self._last_fill_ts.get(token_id, 0.0) < 5.0:
continue
self.set_position(token_id, size, avg)
# ── in-flight guard ─────────────────────────────────────────────────
def mark_inflight(self, token_id: str) -> None:
self._inflight[token_id] = self._inflight.get(token_id, 0) + 1
def clear_inflight(self, token_id: str) -> None:
if self._inflight.get(token_id, 0) > 0:
self._inflight[token_id] -= 1
def inflight(self, token_id: str) -> int:
return self._inflight.get(token_id, 0)
# ── orders ──────────────────────────────────────────────────────────
def orders_for(self, token_id: str) -> list[OpenOrder]:
return [o for o in self.orders.values() if o.token_id == token_id]
def upsert_order(self, order: OpenOrder) -> None:
if order.state in (OrderState.CANCELED, OrderState.DONE, OrderState.REJECTED):
self.orders.pop(order.order_id, None)
else:
self.orders[order.order_id] = order
self._persist_order(order)
def remove_order(self, order_id: str) -> None:
self.orders.pop(order_id, None)
def replace_open_orders(self, token_id: str, live: list[OpenOrder]) -> None:
"""Replace our view of a token's open orders from a REST snapshot."""
for oid in [o.order_id for o in self.orders.values() if o.token_id == token_id]:
self.orders.pop(oid, None)
for o in live:
self.orders[o.order_id] = o
# ── persistence ─────────────────────────────────────────────────────
def _persist_position(self, pos: Position) -> None:
self._conn.execute(
"INSERT OR REPLACE INTO positions(token_id,size,avg_price,updated_ts) VALUES(?,?,?,?)",
(pos.token_id, pos.size, pos.avg_price, time.time()),
)
self._conn.commit()
def _record_fill(self, f: Fill) -> None:
self._conn.execute(
"INSERT OR IGNORE INTO fills(trade_id,token_id,side,price,size,is_maker,ts) VALUES(?,?,?,?,?,?,?)",
(f.trade_id, f.token_id, f.side.value, f.price, f.size, int(f.is_maker), f.ts),
)
self._conn.commit()
def _persist_order(self, o: OpenOrder) -> None:
self._conn.execute(
"INSERT OR REPLACE INTO order_log(order_id,token_id,side,price,size,state,ts) VALUES(?,?,?,?,?,?,?)",
(o.order_id, o.token_id, o.side.value, o.price, o.size, o.state.value, time.time()),
)
self._conn.commit()
def _load(self) -> None:
for row in self._conn.execute("SELECT token_id,size,avg_price FROM positions"):
if row["size"] > 0:
self.positions[row["token_id"]] = Position(
row["token_id"], row["size"], row["avg_price"]
)
# ── reporting ───────────────────────────────────────────────────────
def snapshot(self) -> dict[str, object]:
return {
"positions": {k: json.loads(_pos_json(v)) for k, v in self.positions.items() if v.size > 0},
"open_orders": len(self.orders),
}
def _pos_json(p: Position) -> str:
return json.dumps({"size": round(p.size, 4), "avg_price": round(p.avg_price, 4)})
src/polymaker/state/tracker.py
+100
View File
@@ -0,0 +1,100 @@
"""Order/trade lifecycle processing over the StateStore.
Consumes *normalized* user-stream events (the wire-format extraction lives in
userstream/, so this is unit-testable with synthetic events) and drives the
state machine from docs/scoping/02-architecture.md:
Trade: MATCHED -> MINED -> CONFIRMED
└──────────-> FAILED (roll back the optimistic fill, reconcile)
Because we quote post-only, we are always the maker; `our_side` is our side of
each match. We apply the fill optimistically at MATCHED and reverse it on FAILED.
"""
from __future__ import annotations
from collections.abc import Callable
from dataclasses import dataclass
from polymaker.domain import Fill, OpenOrder, OrderState, Side, TradeState
from polymaker.logging import get_logger
from polymaker.state.store import StateStore
log = get_logger("state.tracker")
@dataclass(frozen=True, slots=True)
class TradeEvent:
token_id: str
our_side: Side
price: float
size: float
trade_id: str
status: TradeState
ts: float
@dataclass(frozen=True, slots=True)
class OrderEvent:
order_id: str
token_id: str
side: Side
price: float
remaining_size: float # original - matched
is_cancel: bool = False
class UserEventProcessor:
"""Applies normalized trade/order events to the store."""
def __init__(
self,
store: StateStore,
on_change: Callable[[str], None] | None = None,
on_fill: Callable[[Fill], None] | None = None,
) -> None:
self._store = store
self._on_change = on_change or (lambda _cid: None)
self._on_fill = on_fill or (lambda _fill: None)
# trade_id -> applied Fill, so FAILED can reverse exactly what we applied
self._applied: dict[str, Fill] = {}
def on_trade(self, ev: TradeEvent, condition_id: str) -> None:
if ev.status is TradeState.MATCHED:
if ev.trade_id in self._applied:
return # idempotent: already counted this match
fill = Fill(ev.token_id, ev.our_side, ev.price, ev.size, ev.trade_id, ev.ts, is_maker=True)
self._store.apply_fill(fill)
self._store.mark_inflight(ev.token_id)
self._applied[ev.trade_id] = fill
self._on_fill(fill)
self._on_change(condition_id)
elif ev.status in (TradeState.CONFIRMED, TradeState.MINED):
if ev.trade_id in self._applied and ev.status is TradeState.CONFIRMED:
self._store.clear_inflight(ev.token_id)
# keep the fill; it's now settled
self._applied.pop(ev.trade_id, None)
self._on_change(condition_id)
elif ev.status in (TradeState.FAILED, TradeState.RETRYING):
prior = self._applied.pop(ev.trade_id, None)
if prior is not None:
# reverse the optimistic fill
self._store.apply_fill(
Fill(prior.token_id, prior.side.opposite, prior.price, prior.size,
f"{prior.trade_id}:reverse", prior.ts, is_maker=True)
)
self._store.clear_inflight(ev.token_id)
log.warning("trade_failed_reversed", trade_id=ev.trade_id, token=ev.token_id[:12])
self._on_change(condition_id)
def on_order(self, ev: OrderEvent, condition_id: str) -> None:
if ev.is_cancel or ev.remaining_size <= 0:
self._store.remove_order(ev.order_id)
else:
state = OrderState.LIVE
self._store.upsert_order(
OpenOrder(ev.order_id, ev.token_id, ev.side, ev.price, ev.remaining_size, state)
)
self._on_change(condition_id)
src/polymaker/strategy/__init__.py
View File
src/polymaker/strategy/estimators.py
+195
View File
@@ -0,0 +1,195 @@
"""Online estimators driven by the live stream: EWMAs of vol, flow, toxicity.
All are time-decayed (half-life in seconds) so they behave correctly under
irregular event arrival — a burst of ticks and a quiet minute are weighted by
elapsed wall-clock, not by sample count. Pure state machines: feed them
observations with timestamps, read scalar summaries. No I/O.
"""
from __future__ import annotations
import math
from dataclasses import dataclass, field
from polymaker.domain import Side
class Ewma:
"""Time-decayed exponentially weighted mean.
On each update the prior weight decays by 0.5 ** (dt / halflife); a fresh
observation gets the remaining weight. The first observation seeds the mean.
"""
__slots__ = ("halflife", "_value", "_last_ts", "_initialized")
def __init__(self, halflife_s: float) -> None:
if halflife_s <= 0:
raise ValueError("halflife must be positive")
self.halflife = halflife_s
self._value = 0.0
self._last_ts = 0.0
self._initialized = False
def update(self, value: float, ts: float) -> float:
if not self._initialized:
self._value = value
self._last_ts = ts
self._initialized = True
return self._value
dt = max(0.0, ts - self._last_ts)
decay = 0.5 ** (dt / self.halflife)
self._value = decay * self._value + (1.0 - decay) * value
self._last_ts = ts
return self._value
def decay_to(self, ts: float) -> float:
"""Decay the stored value toward 0 as if observing 0 at `ts`.
Used to age out flow/vol during silence without a new observation.
"""
if self._initialized:
dt = max(0.0, ts - self._last_ts)
self._value *= 0.5 ** (dt / self.halflife)
self._last_ts = ts
return self._value
@property
def value(self) -> float:
return self._value
@property
def ready(self) -> bool:
return self._initialized
class VolEstimator:
"""Realized volatility at two horizons from fair-value changes."""
__slots__ = ("_short", "_long", "_last_fv", "_last_ts")
def __init__(self, short_halflife_s: float, long_halflife_s: float) -> None:
self._short = Ewma(short_halflife_s)
self._long = Ewma(long_halflife_s)
self._last_fv: float | None = None
self._last_ts = 0.0
def update(self, fv: float, ts: float) -> None:
if self._last_fv is not None:
r = fv - self._last_fv
sq = r * r
self._short.update(sq, ts)
self._long.update(sq, ts)
self._last_fv = fv
self._last_ts = ts
@property
def short(self) -> float:
return math.sqrt(max(0.0, self._short.value))
@property
def long(self) -> float:
return math.sqrt(max(0.0, self._long.value))
@property
def ratio(self) -> float:
"""short/long vol ratio; >1 means recent activity above baseline."""
lo = self.long
return self.short / lo if lo > 1e-9 else 1.0
class FlowEstimator:
"""Signed aggressor flow and its normalized strength (a crude z-score)."""
__slots__ = ("_signed", "_abs")
def __init__(self, halflife_s: float) -> None:
self._signed = Ewma(halflife_s)
self._abs = Ewma(halflife_s)
def update(self, aggressor: Side, size: float, ts: float) -> None:
signed = size if aggressor is Side.BUY else -size
self._signed.update(signed, ts)
self._abs.update(abs(size), ts)
def decay_to(self, ts: float) -> None:
self._signed.decay_to(ts)
self._abs.decay_to(ts)
@property
def signed(self) -> float:
return self._signed.value
@property
def z(self) -> float:
"""Signed flow normalized by average trade magnitude, in ~[-1, 1]+."""
denom = self._abs.value
return self._signed.value / denom if denom > 1e-9 else 0.0
@dataclass(slots=True)
class _PendingMarkout:
fv_at_fill: float
side: Side # our side of the fill (BUY => we bought => adverse if price falls)
due_ts: float
class MarkoutTracker:
"""Measures adverse selection: how fair value moves against us after fills.
For each fill we remember FV-at-fill and, after a horizon, compare to the
then-current FV. Signed so that a *positive* markout means the trade was
good (price moved in our favor) and negative means we got picked off. The
toxicity summary is the magnitude of recent adverse (negative) markout,
which the quoter turns into extra spread / less size.
"""
__slots__ = ("_horizon_s", "_pending", "_markout")
def __init__(self, horizon_s: float = 300.0, ewma_halflife_s: float = 1800.0) -> None:
self._horizon_s = horizon_s
self._pending: list[_PendingMarkout] = []
self._markout = Ewma(ewma_halflife_s)
def record_fill(self, side: Side, fv_at_fill: float, ts: float) -> None:
self._pending.append(_PendingMarkout(fv_at_fill, side, ts + self._horizon_s))
def evaluate(self, fv_now: float, ts: float) -> None:
"""Resolve any markouts whose horizon has elapsed."""
still: list[_PendingMarkout] = []
for p in self._pending:
if ts >= p.due_ts:
move = fv_now - p.fv_at_fill
# if we BOUGHT, a rise is good (+); if we SOLD, a fall is good (+)
signed = move if p.side is Side.BUY else -move
self._markout.update(signed, ts)
else:
still.append(p)
self._pending = still
@property
def markout(self) -> float:
return self._markout.value
@property
def toxicity(self) -> float:
"""Non-negative adverse-selection score (0 when fills are benign)."""
return max(0.0, -self._markout.value)
@dataclass(slots=True)
class MarketEstimators:
"""Bundle of the per-market online estimators the engine keeps."""
vol: VolEstimator
flow: FlowEstimator
markout: MarkoutTracker
last_fv: float | None = None
last_fv_ts: float = 0.0
fv_history: list[tuple[float, float]] = field(default_factory=list)
def on_fair_value(self, fv: float, ts: float) -> None:
self.vol.update(fv, ts)
self.markout.evaluate(fv, ts)
self.last_fv = fv
self.last_fv_ts = ts
src/polymaker/strategy/quoting.py
+203
View File
@@ -0,0 +1,203 @@
"""Pure quote construction: (market state, inventory, params) -> TargetQuotes.
This is the deterministic core of the strategy. No I/O, no wall-clock reads
except values passed in. Everything here is exercised directly by unit tests.
Model (see docs/scoping/04-strategy.md):
reservation r = FV - skew(inventory)
half-spread δ = base + c_vol·σ + c_tox·toxicity (clamped to reward band in QUIET)
YES entry bid = r - δ (BUY YES, USDC-collateralized)
NO entry bid = (1 - r) - δ (BUY NO; implied YES ask at r + δ)
exits = SELL limits on held inventory, walked toward the touch by urgency
The BUY-YES + BUY-NO pair is the canonical two-sided quote: both are bids, both
score rewards, and a filled pair merges back to USDC at locked edge 1 - p - q.
"""
from __future__ import annotations
import math
from dataclasses import dataclass
from polymaker.config import StrategyProfile
from polymaker.domain import MarketMeta, Position, Quote, Regime, Side, TargetQuotes
from polymaker.marketdata.orderbook import BookView
_EPS = 1e-9
def round_to_tick(price: float, tick: float, decimals: int, *, up: bool) -> float:
"""Snap a price to the tick grid, rounding up or down, clamped to (0,1)."""
n = price / tick
n = math.ceil(n - _EPS) if up else math.floor(n + _EPS)
p = round(n * tick, decimals)
return min(max(p, tick), 1.0 - tick)
def compute_fair_value(microprice: float, flow_z: float, tick: float, weight: float = 0.5) -> float:
"""Nudge the microprice by bounded signed flow. Clamped to (tick, 1-tick)."""
fv = microprice + weight * flow_z * tick
return min(max(fv, tick), 1.0 - tick)
@dataclass(frozen=True, slots=True)
class QuoteInputs:
meta: MarketMeta
regime: Regime
fv: float # YES fair value in (0,1)
vol_short: float
toxicity: float
yes_view: BookView
no_view: BookView
pos_yes: Position
pos_no: Position
profile: StrategyProfile
now: float
risk_size_scale: float = 1.0 # RiskManager may throttle size in [0,1]
yes_exit_urgency: float = 0.0 # [0,1]; engine raises with hold time / adverse drift
no_exit_urgency: float = 0.0
def construct_quotes(inp: QuoteInputs) -> TargetQuotes:
m = inp.meta
p = inp.profile
tick = m.tick_size
dec = m.price_decimals
cid = m.condition_id
if inp.regime in (Regime.EVENT, Regime.HALTED):
return TargetQuotes(cid, inp.regime, ())
quotes: list[Quote] = []
# ── inventory in YES-equivalent shares; holding NO is short YES ──────
net_shares = inp.pos_yes.size - inp.pos_no.size
q_max_shares = p.q_max_usdc / max(inp.fv, tick)
u = _clamp(net_shares / q_max_shares, -1.0, 1.0) if q_max_shares > 0 else 0.0
skew = p.gamma * inp.vol_short * u
# ── half-spread ─────────────────────────────────────────────────────
base = p.delta_min_ticks * tick
delta = base + p.c_vol * inp.vol_short + p.c_tox * inp.toxicity
reward_band = m.rewards_max_spread / 100.0
if inp.regime == Regime.QUIET and reward_band > 0:
delta = _clamp(delta, base, max(base, reward_band))
delta = max(delta, tick)
r = inp.fv - skew
yes_bid_target = r - delta
no_bid_target = (1.0 - r) - delta
# ── size scaling ────────────────────────────────────────────────────
regime_scale = 0.5 if inp.regime == Regime.TRENDING else 1.0
tox_scale = 1.0 / (1.0 + inp.toxicity * 10.0)
common_scale = regime_scale * tox_scale * _clamp(inp.risk_size_scale, 0.0, 1.0)
soft_cap = p.q_soft_frac # fraction of q_max at which the adding side pulls
add_yes = inp.regime not in (Regime.REDUCE_ONLY,) and u < soft_cap
add_no = inp.regime not in (Regime.REDUCE_ONLY,) and u > -soft_cap
# entry: BUY YES
if add_yes:
price = _place_bid(yes_bid_target, inp.yes_view, tick, dec, inp.fv, p.min_edge_ticks)
if price is not None:
_add_layers(quotes, m.yes.token_id, Side.BUY, price, tick, dec,
_size_shares(p.base_size_usdc, price, common_scale * (1 - max(u, 0.0)), m),
p.layers, p.layer_step_ticks, down=True)
# entry: BUY NO
if add_no:
no_fv = 1.0 - inp.fv
price = _place_bid(no_bid_target, inp.no_view, tick, dec, no_fv, p.min_edge_ticks)
if price is not None:
_add_layers(quotes, m.no.token_id, Side.BUY, price, tick, dec,
_size_shares(p.base_size_usdc, price, common_scale * (1 - max(-u, 0.0)), m),
p.layers, p.layer_step_ticks, down=True)
# ── exits: SELL held inventory (maker, never cross) ─────────────────
_maybe_exit(quotes, m.yes.token_id, inp.pos_yes, inp.fv, delta, inp.yes_view, tick, dec,
inp.yes_exit_urgency, m, inp.regime)
_maybe_exit(quotes, m.no.token_id, inp.pos_no, 1.0 - inp.fv, delta, inp.no_view, tick, dec,
inp.no_exit_urgency, m, inp.regime)
return TargetQuotes(cid, inp.regime, tuple(quotes))
# ── helpers ─────────────────────────────────────────────────────────────
def _clamp(x: float, lo: float, hi: float) -> float:
return min(max(x, lo), hi)
def _place_bid(
target: float, view: BookView, tick: float, dec: int, fv: float, min_edge_ticks: int
) -> float | None:
"""Position a BUY: join the touch or sit behind, never cross, keep min edge vs FV."""
price = target
# never bid above (FV - min_edge*tick): we don't pay through fair value
price = min(price, fv - min_edge_ticks * tick)
# join the queue rather than jump it (conservative maker default)
if view.best_bid is not None and price >= view.best_bid:
price = view.best_bid
# never cross the ask
if view.best_ask is not None and price >= view.best_ask:
price = view.best_ask - tick
p = round_to_tick(price, tick, dec, up=False)
if p <= 0 or p >= 1:
return None
return p
def _size_shares(base_usdc: float, price: float, scale: float, m: MarketMeta) -> float:
"""USDC-notional sizing -> shares, honoring exchange & reward minimums."""
shares = (base_usdc / max(price, m.tick_size)) * max(scale, 0.0)
if shares <= 0:
return 0.0
floor = max(m.min_order_size, m.rewards_min_size)
# round up small-but-real sizes to the reward min so they actually score
if 0.5 * floor <= shares < floor:
shares = floor
return round(shares, 2) if shares >= m.min_order_size else 0.0
def _add_layers(
quotes: list[Quote], token_id: str, side: Side, top_price: float, tick: float, dec: int,
total_size: float, layers: int, step_ticks: int, *, down: bool,
) -> None:
"""Split size across `layers` price levels stepping away from the touch."""
if total_size <= 0:
return
layers = max(1, layers)
per = round(total_size / layers, 2)
if per <= 0:
per = total_size
layers = 1
for i in range(layers):
offset = i * step_ticks * tick
price = top_price - offset if down else top_price + offset
price = round(price, dec)
if 0 < price < 1 and per > 0:
quotes.append(Quote(token_id, side, price, per))
def _maybe_exit(
quotes: list[Quote], token_id: str, pos: Position, token_fv: float, delta: float,
view: BookView, tick: float, dec: int, urgency: float, m: MarketMeta, regime: Regime,
) -> None:
if pos.size < m.min_order_size:
return
# target starts at fv + delta and walks toward best_bid + tick as urgency -> 1
passive = token_fv + delta
floor = (view.best_bid + tick) if view.best_bid is not None else passive
if regime == Regime.REDUCE_ONLY:
urgency = max(urgency, 0.5)
target = passive * (1.0 - urgency) + floor * urgency
# never cross down through the bid; never sell below best_bid
if view.best_bid is not None:
target = max(target, view.best_bid + tick)
price = round_to_tick(target, tick, dec, up=True)
size = round(pos.size, 2)
if 0 < price < 1 and size >= m.min_order_size:
quotes.append(Quote(token_id, Side.SELL, price, size))
src/polymaker/strategy/regime.py
+74
View File
@@ -0,0 +1,74 @@
"""Per-market regime decision (see docs/scoping/04-strategy.md §5).
Priority order, highest first:
HALTED kill switch / stale data / resolved / past halt-before window
EVENT active cooloff, or a fresh sweep / fair-value jump
REDUCE_ONLY inventory at hard cap, or inside the reduce-only end-date window
TRENDING persistent one-sided flow or elevated short/long vol
QUIET default farming posture
"""
from __future__ import annotations
from dataclasses import dataclass
from polymaker.config import StrategyProfile
from polymaker.domain import Regime
@dataclass(frozen=True, slots=True)
class RegimeInputs:
now: float
tick: float
fv: float
prev_fv: float | None
vol_ratio: float
flow_z: float
inventory_util: float # |net notional| / q_max, >=0
hours_to_end: float | None
sweep_flagged: bool = False
market_resolved: bool = False
ws_stale: bool = False
risk_halt: bool = False
risk_reduce_only: bool = False
class RegimeMachine:
"""Stateful regime decider for one market (tracks the EVENT cooloff)."""
__slots__ = ("_event_until",)
def __init__(self) -> None:
self._event_until: float = 0.0
def decide(self, inp: RegimeInputs, p: StrategyProfile) -> Regime:
# 1. hard halts
if inp.risk_halt or inp.ws_stale or inp.market_resolved:
return Regime.HALTED
if inp.hours_to_end is not None and inp.hours_to_end <= p.halt_before_hours:
return Regime.HALTED
# 2. events (sweep / jump / active cooloff)
jump_ticks = abs(inp.fv - inp.prev_fv) / inp.tick if inp.prev_fv is not None else 0.0
if inp.sweep_flagged or jump_ticks >= p.event_jump_ticks:
self._event_until = inp.now + p.event_cooloff_s
return Regime.EVENT
if inp.now < self._event_until:
return Regime.EVENT
# 3. reduce-only
if inp.risk_reduce_only or inp.inventory_util >= 1.0:
return Regime.REDUCE_ONLY
if inp.hours_to_end is not None and inp.hours_to_end <= p.reduce_only_hours:
return Regime.REDUCE_ONLY
# 4. trending
if abs(inp.flow_z) >= p.trend_flow_z or inp.vol_ratio >= 2.0:
return Regime.TRENDING
# 5. default
return Regime.QUIET
@property
def in_cooloff(self) -> bool:
return self._event_until > 0.0
src/polymaker/userstream/__init__.py
View File
src/polymaker/userstream/client.py
+126
View File
@@ -0,0 +1,126 @@
"""UserStream: authenticated user WS for our order/trade lifecycle events.
Subscribes with L2 creds and the condition_ids we trade; routes fills and order
updates into the StateStore via the UserEventProcessor. Reconnects with backoff.
"""
from __future__ import annotations
import asyncio
import json
from collections.abc import Callable
from typing import Any
import websockets
from polymaker.journal import Journal
from polymaker.logging import get_logger
from polymaker.state.tracker import UserEventProcessor
from polymaker.userstream.parse import normalize_order, normalize_trade
log = get_logger("userstream.client")
class UserStream:
def __init__(
self,
creds: Any,
our_address: str,
processor: UserEventProcessor,
*,
other_token: Callable[[str], str | None],
condition_of_token: Callable[[str], str | None],
url: str = "wss://ws-subscriptions-clob.polymarket.com/ws/user",
journal: Journal | None = None,
proxy: str | None = None,
) -> None:
self._creds = creds
self._address = our_address
self._proc = processor
self._other_token = other_token
self._condition_of_token = condition_of_token
self._url = url
self._journal = journal
self._proxy = proxy
self._markets: list[str] = []
self._stop = asyncio.Event()
def set_markets(self, condition_ids: list[str]) -> None:
self._markets = condition_ids
async def run(self) -> None:
backoff = 1.0
while not self._stop.is_set():
try:
await self._connect_and_listen()
backoff = 1.0
except (websockets.ConnectionClosed, OSError) as exc:
log.warning("user_ws_dropped", err=str(exc))
except Exception as exc: # noqa: BLE001
log.error("user_ws_error", err=str(exc))
if self._stop.is_set():
break
await asyncio.sleep(backoff)
backoff = min(backoff * 2, 30.0)
async def _connect_and_listen(self) -> None:
sub = {
"type": "user",
"auth": {
"apiKey": self._creds.api_key,
"secret": self._creds.api_secret,
"passphrase": self._creds.api_passphrase,
},
"markets": self._markets,
}
kwargs: dict[str, Any] = {"ping_interval": 5, "ping_timeout": None}
if self._proxy:
kwargs["proxy"] = self._proxy
async with websockets.connect(self._url, **kwargs) as ws:
await ws.send(json.dumps(sub))
log.info("user_ws_subscribed", markets=len(self._markets))
async for raw in ws:
self._handle(raw)
def stop(self) -> None:
self._stop.set()
def _handle(self, raw: str | bytes) -> None:
try:
data = json.loads(raw)
except (json.JSONDecodeError, TypeError):
return
for msg in data if isinstance(data, list) else [data]:
if not isinstance(msg, dict):
continue
et = msg.get("event_type")
if et == "trade":
self._on_trade(msg)
elif et == "order":
self._on_order(msg)
def _on_trade(self, msg: dict[str, Any]) -> None:
self._journal_write("user_trade", msg)
for ev in normalize_trade(msg, self._address, self._other_token):
cond = self._condition_of_token(ev.token_id) or str(msg.get("market", ""))
self._proc.on_trade(ev, cond)
def _on_order(self, msg: dict[str, Any]) -> None:
self._journal_write("user_order", msg)
ev = normalize_order(msg)
if ev is not None:
cond = self._condition_of_token(ev.token_id) or str(msg.get("market", ""))
self._proc.on_order(ev, cond)
def _journal_write(self, kind: str, payload: dict[str, Any]) -> None:
if self._journal is not None:
self._journal.write(kind, payload, _ts(payload))
def _ts(msg: dict[str, Any]) -> float:
raw = msg.get("timestamp")
try:
v = float(raw) # type: ignore[arg-type]
return v / 1000.0 if v > 1e12 else v
except (ValueError, TypeError):
return 0.0
src/polymaker/userstream/parse.py
+111
View File
@@ -0,0 +1,111 @@
"""Pure parsers for user-WS frames -> normalized trade/order events.
Fills come from `trade` events; open-order tracking from `order` events. The
maker/taker/mint side logic mirrors v1's proven handler (post-only means we are
always the maker):
* maker & taker on the SAME outcome -> we SELL the taker's asset (reverse side)
* maker & taker on DIFFERENT outcomes -> a mint: we BUY the opposite token
NOTE: exact field names must be reconfirmed in the Phase-2 wallet spike
(docs/scoping/03-api-layer.md §9); this is coded to the v1-observed shape.
"""
from __future__ import annotations
from collections.abc import Callable
from typing import Any
from polymaker.domain import Side, TradeState
from polymaker.state.tracker import OrderEvent, TradeEvent
_STATUS = {
"MATCHED": TradeState.MATCHED,
"MINED": TradeState.MINED,
"CONFIRMED": TradeState.CONFIRMED,
"RETRYING": TradeState.RETRYING,
"FAILED": TradeState.FAILED,
}
def _ts(msg: dict[str, Any]) -> float:
raw = msg.get("timestamp")
try:
v = float(raw) # type: ignore[arg-type]
return v / 1000.0 if v > 1e12 else v
except (ValueError, TypeError):
return 0.0
def normalize_trade(
msg: dict[str, Any],
our_address: str,
other_token: Callable[[str], str | None],
) -> list[TradeEvent]:
"""Extract our maker fills from a `trade` event. Returns one TradeEvent per
matching maker order (usually one)."""
status = _STATUS.get(str(msg.get("status", "")).upper())
if status is None:
return []
taker_asset = str(msg.get("asset_id", ""))
taker_side = _side(msg.get("side"))
taker_outcome = msg.get("outcome")
ts = _ts(msg)
trade_id = str(msg.get("id", ""))
addr = our_address.lower()
out: list[TradeEvent] = []
for i, mo in enumerate(msg.get("maker_orders", []) or []):
if str(mo.get("maker_address", "")).lower() != addr:
continue
try:
size = float(mo.get("matched_amount", 0))
price = float(mo.get("price", 0))
except (ValueError, TypeError):
continue
if size <= 0:
continue
if mo.get("outcome") == taker_outcome:
token = taker_asset
our_side = taker_side.opposite
else:
token = other_token(taker_asset) or taker_asset
our_side = taker_side
out.append(
TradeEvent(
token_id=token,
our_side=our_side,
price=price,
size=size,
trade_id=f"{trade_id}:{i}" if len(msg.get('maker_orders', [])) > 1 else trade_id,
status=status,
ts=ts,
)
)
return out
def normalize_order(msg: dict[str, Any]) -> OrderEvent | None:
"""Map an `order` event to remaining-size tracking for the reconciler."""
try:
asset = str(msg["asset_id"])
side = _side(msg.get("side"))
original = float(msg.get("original_size", msg.get("size", 0)))
matched = float(msg.get("size_matched", 0))
remaining = original - matched
status = str(msg.get("status", "")).upper()
is_cancel = status in ("CANCELED", "CANCELLED") or msg.get("type") == "CANCELLATION"
return OrderEvent(
order_id=str(msg.get("id", "")),
token_id=asset,
side=side,
price=float(msg.get("price", 0)),
remaining_size=remaining,
is_cancel=is_cancel,
)
except (KeyError, ValueError, TypeError):
return None
def _side(value: object) -> Side:
return Side.SELL if str(value).upper() == "SELL" else Side.BUY
tests/__init__.py
View File
tests/conftest.py
+49
View File
@@ -0,0 +1,49 @@
"""Shared test fixtures."""
from __future__ import annotations
import pytest
from polymaker.config import StrategyProfile
from polymaker.domain import MarketMeta, TokenMeta
from polymaker.marketdata.orderbook import BookView
@pytest.fixture
def meta() -> MarketMeta:
return MarketMeta(
condition_id="0xcond",
question="Will X happen?",
slug="will-x-happen",
tokens=(TokenMeta("yes-token", "Yes"), TokenMeta("no-token", "No")),
tick_size=0.01,
neg_risk=False,
min_order_size=5.0,
rewards_min_size=10.0,
rewards_max_spread=3.0, # 3 cents
rewards_daily_rate=50.0,
maker_fee_bps=0,
taker_fee_bps=100,
fees_enabled=True,
end_date_iso="2028-11-07T00:00:00Z",
event_id="evt-1",
)
@pytest.fixture
def profile() -> StrategyProfile:
return StrategyProfile() # defaults
def view(bb: float | None, ba: float | None, bb_sz: float = 500, ba_sz: float = 500) -> BookView:
"""Construct a BookView for a token with a symmetric deep book."""
return BookView(
best_bid=bb,
best_bid_size=bb_sz,
best_ask=ba,
best_ask_size=ba_sz,
second_bid=(bb - 0.01) if bb else None,
second_ask=(ba + 0.01) if ba else None,
bid_depth=bb_sz,
ask_depth=ba_sz,
)
tests/test_catalog.py
+91
View File
@@ -0,0 +1,91 @@
"""Tests for market parsing, scoring, and the SQLite catalog store."""
from __future__ import annotations
import json
from polymaker.catalog.gamma import parse_market
from polymaker.catalog.scoring import score_market
from polymaker.catalog.store import CatalogStore
RAW = {
"conditionId": "0xabc",
"question": "Will candidate X win?",
"slug": "will-x-win",
"clobTokenIds": json.dumps(["tok-yes", "tok-no"]),
"outcomes": json.dumps(["Yes", "No"]),
"orderPriceMinTickSize": 0.01,
"orderMinSize": 5,
"negRisk": True,
"acceptingOrders": True,
"rewardsMinSize": 10,
"rewardsMaxSpread": 3.0,
"feesEnabled": True,
"feeSchedule": {"rate": 0.01, "takerOnly": True, "rebateRate": 0.25},
"bestBid": 0.48,
"bestAsk": 0.50,
"liquidityNum": 20000.0,
"volumeNum": 500000.0,
"endDate": "2028-11-07T00:00:00Z",
"events": [{"id": 999, "slug": "2028-election"}],
}
def test_parse_market_maps_fields():
m = parse_market(RAW, reward_rates={"0xabc": 42.0})
assert m is not None
assert m.condition_id == "0xabc"
assert m.yes.token_id == "tok-yes"
assert m.no.token_id == "tok-no"
assert m.tick_size == 0.01
assert m.neg_risk is True
assert m.rewards_daily_rate == 42.0
assert m.taker_fee_bps == 100 # 0.01 -> 100 bps
assert m.maker_fee_bps == 0 # V2 makers pay zero
assert m.rebate_rate == 0.25
assert m.event_id == "999"
def test_parse_market_rejects_non_binary_and_closed():
triple = {**RAW, "clobTokenIds": json.dumps(["a", "b", "c"]),
"outcomes": json.dumps(["A", "B", "C"])}
assert parse_market(triple) is None
not_accepting = {**RAW, "acceptingOrders": False}
assert parse_market(not_accepting) is None
def test_score_prefers_rewards_and_rebates():
good = parse_market(RAW, {"0xabc": 100.0})
poor = parse_market({**RAW, "conditionId": "0xdef", "rewardsMinSize": 0,
"rewardsMaxSpread": 0, "feesEnabled": False},
{"0xdef": 0.0})
assert score_market(good).score > score_market(poor).score
def test_score_penalizes_extremity():
balanced = parse_market(RAW, {"0xabc": 50.0})
extreme = parse_market({**RAW, "conditionId": "0xext", "bestBid": 0.96, "bestAsk": 0.98},
{"0xext": 50.0})
assert score_market(extreme).extremity > score_market(balanced).extremity
def test_store_roundtrip_and_top(tmp_path):
store = CatalogStore(tmp_path / "s.db")
m = parse_market(RAW, {"0xabc": 42.0})
store.upsert_market(m)
assert store.get("0xabc").condition_id == "0xabc"
assert store.get_by_slug("will-x-win").slug == "will-x-win"
top = store.top(10)
assert len(top) == 1 and top[0][0].condition_id == "0xabc"
# tokens survive the JSON round-trip as a 2-tuple
assert len(store.get("0xabc").tokens) == 2
store.close()
def test_store_upsert_is_idempotent(tmp_path):
store = CatalogStore(tmp_path / "s.db")
m = parse_market(RAW, {"0xabc": 42.0})
store.upsert_market(m)
store.upsert_market(m) # second time updates, not duplicates
assert len(store.top(10)) == 1
store.close()
tests/test_engine.py
+76
View File
@@ -0,0 +1,76 @@
"""Integration test: one full engine recompute cycle in paper mode (no network)."""
from __future__ import annotations
import asyncio
import time
from polymaker.config import Config, PathsConfig, StrategyProfile
from polymaker.domain import Side
from polymaker.engine import Engine
from polymaker.strategy.regime import RegimeMachine
def _engine_with_market(tmp_path, meta) -> Engine:
cfg = Config(paths=PathsConfig(db=str(tmp_path / "state.db"),
journal_dir=str(tmp_path / "j"),
log_dir=str(tmp_path / "l")))
cfg.engine.journal = False
eng = Engine(cfg, paper=True)
cid = meta.condition_id
# inject one market directly, bypassing network resolution
eng.metas[cid] = meta
eng.profiles[cid] = StrategyProfile()
eng.est[cid] = Engine._make_estimators(eng.profiles[cid])
eng.regime_m[cid] = RegimeMachine()
eng._dirty[cid] = asyncio.Event()
for tok in (meta.yes.token_id, meta.no.token_id):
eng._token_cid[tok] = cid
eng.md.set_markets([(cid, [meta.yes.token_id, meta.no.token_id])])
eng._running = True
return eng
def _feed_book(eng, meta):
now = time.time() # fresh ts so the ws_stale guard doesn't HALT the market
yb = eng.md.book(meta.yes.token_id)
yb.apply_snapshot(bids=[(0.48, 500), (0.49, 500)], asks=[(0.51, 500), (0.52, 500)], ts=now)
nb = eng.md.book(meta.no.token_id)
nb.apply_snapshot(bids=[(0.48, 500), (0.49, 500)], asks=[(0.51, 500), (0.52, 500)], ts=now)
async def test_recompute_places_two_sided_paper_quotes(tmp_path, meta):
eng = _engine_with_market(tmp_path, meta)
_feed_book(eng, meta)
await eng._recompute(meta.condition_id)
yes_orders = eng.state.orders_for(meta.yes.token_id)
no_orders = eng.state.orders_for(meta.no.token_id)
assert yes_orders, "no YES quotes placed"
assert no_orders, "no NO quotes placed"
# entry quotes are BUYs on both tokens (the canonical two-sided quote)
assert all(o.side is Side.BUY for o in yes_orders)
assert all(o.side is Side.BUY for o in no_orders)
eng.state.close()
eng.catalog.close()
async def test_recompute_is_idempotent_within_tolerance(tmp_path, meta):
eng = _engine_with_market(tmp_path, meta)
_feed_book(eng, meta)
await eng._recompute(meta.condition_id)
n_after_first = len(eng.state.orders)
# same book -> reconcile should be a no-op, order count unchanged
await eng._recompute(meta.condition_id)
assert len(eng.state.orders) == n_after_first
eng.state.close()
eng.catalog.close()
async def test_recompute_skips_when_book_empty(tmp_path, meta):
eng = _engine_with_market(tmp_path, meta)
# no book fed
await eng._recompute(meta.condition_id)
assert len(eng.state.orders) == 0
eng.state.close()
eng.catalog.close()
tests/test_estimators.py
+83
View File
@@ -0,0 +1,83 @@
"""Unit tests for the online estimators (vol, flow, markout/toxicity)."""
from __future__ import annotations
import pytest
from polymaker.domain import Side
from polymaker.strategy.estimators import (
Ewma,
FlowEstimator,
MarkoutTracker,
VolEstimator,
)
def test_ewma_seeds_then_decays():
e = Ewma(halflife_s=10.0)
assert not e.ready
e.update(1.0, ts=0.0)
assert e.value == 1.0
# after exactly one half-life, a 0 observation pulls the mean halfway
e.update(0.0, ts=10.0)
assert e.value == pytest.approx(0.5, abs=1e-9)
def test_ewma_decay_to_ages_value():
e = Ewma(halflife_s=10.0)
e.update(1.0, ts=0.0)
e.decay_to(ts=10.0) # one half-life of silence
assert e.value == pytest.approx(0.5, abs=1e-9)
def test_vol_estimator_rises_with_movement():
v = VolEstimator(short_halflife_s=5.0, long_halflife_s=100.0)
# quiet: tiny moves
fv, t = 0.5, 0.0
for _ in range(20):
t += 1.0
v.update(fv, t) # no change -> zero vol
assert v.short == pytest.approx(0.0, abs=1e-6)
# sudden jumps -> short vol jumps, ratio > 1
for step in (0.05, -0.04, 0.06):
t += 1.0
fv += step
v.update(fv, t)
assert v.short > 0.01
assert v.ratio > 1.0
def test_flow_estimator_sign_and_z():
f = FlowEstimator(halflife_s=10.0)
t = 0.0
for _ in range(5):
t += 1.0
f.update(Side.BUY, 100, t) # persistent buying
assert f.signed > 0
assert f.z > 0.5 # strongly one-sided
# now heavy selling flips the sign over time
for _ in range(10):
t += 1.0
f.update(Side.SELL, 200, t)
assert f.signed < 0
assert f.z < 0
def test_markout_toxicity_from_adverse_fills():
mt = MarkoutTracker(horizon_s=30.0, ewma_halflife_s=100.0)
# we BUY at fv=0.50; price then falls to 0.45 after the horizon -> adverse
mt.record_fill(Side.BUY, fv_at_fill=0.50, ts=0.0)
mt.evaluate(fv_now=0.50, ts=10.0) # before horizon: nothing resolves
assert mt.markout == 0.0
mt.evaluate(fv_now=0.45, ts=31.0) # after horizon: -0.05 markout
assert mt.markout < 0
assert mt.toxicity > 0
def test_markout_benign_fills_are_not_toxic():
mt = MarkoutTracker(horizon_s=30.0, ewma_halflife_s=100.0)
# we BUY at 0.50; price rises to 0.55 -> favorable, not toxic
mt.record_fill(Side.BUY, fv_at_fill=0.50, ts=0.0)
mt.evaluate(fv_now=0.55, ts=31.0)
assert mt.markout > 0
assert mt.toxicity == 0.0
tests/test_execution.py
+69
View File
@@ -0,0 +1,69 @@
"""Tests for the rate budgeter and the paper-mode gateway."""
from __future__ import annotations
import asyncio
import time
import pytest
from polymaker.config import Config
from polymaker.domain import Quote, Side
from polymaker.execution.gateway import ExecutionGateway, _tick_str
from polymaker.execution.ratelimit import TokenBucket
def test_tick_str_formats():
assert _tick_str(0.01) == "0.01"
assert _tick_str(0.001) == "0.001"
assert _tick_str(0.0025) == "0.0025"
assert _tick_str(0.1) == "0.1"
async def test_token_bucket_limits_rate():
bucket = TokenBucket(rate_per_s=100.0, burst=5.0)
start = time.monotonic()
# burst of 5 is instant; the next 5 must wait ~ (5/100)s = 50ms
for _ in range(10):
await bucket.acquire(1)
elapsed = time.monotonic() - start
assert elapsed >= 0.04 # had to wait for refill
async def test_token_bucket_pressure_rises_when_drained():
bucket = TokenBucket(rate_per_s=10.0, burst=10.0)
assert bucket.pressure == pytest.approx(0.0, abs=0.01)
for _ in range(10):
await bucket.acquire(1)
assert bucket.pressure > 0.8
async def test_paper_gateway_places_and_cancels_without_wallet(meta):
cfg = Config() # defaults, no secrets
gw = ExecutionGateway(cfg, paper=True)
quotes = [
Quote(meta.yes.token_id, Side.BUY, 0.49, 100),
Quote(meta.no.token_id, Side.BUY, 0.48, 100),
]
placed = await gw.place(quotes, meta)
assert len(placed) == 2
assert all(o.order_id.startswith("paper-") for o in placed)
# cancel is a no-op in paper mode but must not raise
await gw.cancel([o.order_id for o in placed])
assert await gw.open_orders() == []
async def test_paper_gateway_heartbeat_and_cancel_all_noop():
gw = ExecutionGateway(Config(), paper=True)
await gw.heartbeat("hb1")
await gw.cancel_all() # no client, must not raise
def test_gateway_requires_wallet_for_live_connect():
from polymaker.config import Secrets
# explicitly-empty secrets (don't read a real .env that may exist on disk)
cfg = Config(secrets=Secrets(_env_file=None))
gw = ExecutionGateway(cfg, paper=False)
with pytest.raises(RuntimeError, match="no wallet"):
asyncio.run(gw.connect())
tests/test_live_marketdata.py
+74
View File
@@ -0,0 +1,74 @@
"""Live integration test for the market WS (network-gated).
Run explicitly with: POLYMAKER_LIVE=1 uv run pytest tests/test_live_marketdata.py
Skipped by default so the unit suite stays offline and fast.
"""
from __future__ import annotations
import asyncio
import json
import os
import httpx
import pytest
import websockets
pytestmark = pytest.mark.skipif(
os.environ.get("POLYMAKER_LIVE") != "1", reason="live test; set POLYMAKER_LIVE=1"
)
async def _top_political_tokens() -> tuple[str, list[str]]:
async with httpx.AsyncClient(timeout=20) as c:
r = await c.get(
"https://gamma-api.polymarket.com/markets",
params={"limit": 1, "closed": "false", "tag_id": 2,
"order": "volume24hr", "ascending": "false"},
)
m = r.json()[0]
return m["conditionId"], json.loads(m["clobTokenIds"])
async def test_market_service_builds_a_live_book():
from polymaker.marketdata.service import MarketDataService
cond, tokens = await _top_political_tokens()
woken: list[tuple[str, str]] = []
svc = MarketDataService(on_dirty=lambda c, t: woken.append((c, t)))
svc.set_markets([(cond, tokens)])
task = asyncio.create_task(svc.run())
try:
# wait until at least one token has a two-sided book
for _ in range(60):
await asyncio.sleep(0.5)
if any(not svc.book(t).is_empty for t in tokens):
break
finally:
svc.stop()
task.cancel()
assert woken, "quoter was never woken by a book event"
live = [t for t in tokens if not svc.book(t).is_empty]
assert live, "no book was populated from the live feed"
book = svc.book(live[0])
assert book.best_bid() is not None and book.best_ask() is not None
assert book.best_bid().price < book.best_ask().price
async def test_live_market_ws_raw_frames():
"""Sanity check the wire format our parser targets hasn't drifted."""
_, tokens = await _top_political_tokens()
uri = "wss://ws-subscriptions-clob.polymarket.com/ws/market"
async with websockets.connect(uri, ping_interval=5, ping_timeout=None) as ws:
await ws.send(json.dumps({"assets_ids": tokens, "type": "market"}))
got_book = False
for _ in range(10):
raw = await asyncio.wait_for(ws.recv(), timeout=10)
data = json.loads(raw)
for msg in data if isinstance(data, list) else [data]:
if msg.get("event_type") == "book":
assert {"asset_id", "bids", "asks", "hash"} <= set(msg)
got_book = True
assert got_book
tests/test_marketdata.py
+110
View File
@@ -0,0 +1,110 @@
"""Tests for market-WS parsing and the book service routing."""
from __future__ import annotations
from polymaker.domain import Side
from polymaker.marketdata.parse import (
parse_book,
parse_last_trade,
parse_price_changes,
parse_tick_size_change,
)
from polymaker.marketdata.service import MarketDataService
# frames modeled on live captures (2026-07-05)
BOOK = {
"event_type": "book",
"market": "0xcond",
"asset_id": "yes-tok",
"timestamp": "1783270000000",
"hash": "abc123",
"tick_size": "0.01",
"bids": [{"price": "0.48", "size": "100"}, {"price": "0.49", "size": "200"}],
"asks": [{"price": "0.52", "size": "150"}, {"price": "0.51", "size": "80"}],
}
PRICE_CHANGE = {
"event_type": "price_change",
"market": "0xcond",
"timestamp": "1783270001000",
"price_changes": [
{"asset_id": "yes-tok", "price": "0.49", "size": "0", "side": "BUY", "hash": "h"},
{"asset_id": "yes-tok", "price": "0.50", "size": "300", "side": "BUY", "hash": "h"},
],
}
LAST_TRADE = {
"event_type": "last_trade_price",
"market": "0xcond",
"asset_id": "yes-tok",
"price": "0.50",
"size": "42",
"side": "BUY",
"timestamp": "1783270002000",
}
def test_parse_book_converts_ms_and_levels():
upd = parse_book(BOOK)
assert upd is not None
assert upd.asset_id == "yes-tok"
assert upd.condition_id == "0xcond"
assert (0.49, 200) in upd.bids
assert upd.ts == 1783270000.0 # ms -> s
assert upd.tick_size == 0.01
def test_parse_price_changes():
changes = parse_price_changes(PRICE_CHANGE)
assert len(changes) == 2
assert changes[0].side is Side.BUY
assert changes[1].price == 0.50 and changes[1].size == 300
def test_parse_last_trade_aggressor():
tp = parse_last_trade(LAST_TRADE)
assert tp is not None
assert tp.aggressor is Side.BUY
assert tp.size == 42
def test_parse_tick_size_change():
tc = parse_tick_size_change(
{"event_type": "tick_size_change", "asset_id": "yes-tok", "new_tick_size": "0.001"}
)
assert tc is not None and tc.tick_size == 0.001
def test_service_routes_book_and_wakes_quoter():
woken: list[tuple[str, str]] = []
svc = MarketDataService(on_dirty=lambda c, t: woken.append((c, t)))
svc.set_markets([("0xcond", ["yes-tok", "no-tok"])])
svc._dispatch(BOOK)
book = svc.book("yes-tok")
assert book is not None
assert book.best_bid().price == 0.49
assert book.best_ask().price == 0.51
assert woken == [("0xcond", "yes-tok")]
def test_service_applies_price_change_delta():
svc = MarketDataService()
svc.set_markets([("0xcond", ["yes-tok"])])
svc._dispatch(BOOK)
svc._dispatch(PRICE_CHANGE) # removes 0.49 bid, adds 0.50 bid
book = svc.book("yes-tok")
assert book.best_bid().price == 0.50
assert 0.49 not in book.bids
def test_service_ignores_unsubscribed_asset():
svc = MarketDataService()
svc.set_markets([("0xcond", ["yes-tok"])])
svc._dispatch({**BOOK, "asset_id": "stranger"})
assert svc.book("stranger") is None
def test_service_forwards_trades_for_flow():
trades = []
svc = MarketDataService(on_trade=trades.append)
svc.set_markets([("0xcond", ["yes-tok"])])
svc._dispatch(LAST_TRADE)
assert len(trades) == 1 and trades[0].aggressor is Side.BUY
tests/test_orderbook.py
+94
View File
@@ -0,0 +1,94 @@
"""Unit tests for the order book and its analytics."""
from __future__ import annotations
import pytest
from polymaker.domain import Side
from polymaker.marketdata.orderbook import OrderBook, to_no_price
def make_book() -> OrderBook:
ob = OrderBook(tick_size=0.01)
ob.apply_snapshot(
bids=[(0.40, 100), (0.41, 200), (0.42, 50)], # best bid 0.42
asks=[(0.45, 80), (0.46, 150), (0.44, 30)], # best ask 0.44
ts=1.0,
)
return ob
def test_best_bid_ask():
ob = make_book()
assert ob.best_bid().price == 0.42
assert ob.best_bid().size == 50
assert ob.best_ask().price == 0.44
assert ob.best_ask().size == 30
def test_apply_delta_add_and_remove():
ob = make_book()
ob.apply_delta(Side.BUY, 0.43, 25, ts=2.0)
assert ob.best_bid().price == 0.43
ob.apply_delta(Side.BUY, 0.43, 0, ts=3.0) # size 0 removes the level
assert ob.best_bid().price == 0.42
assert ob.last_update_ts == 3.0
def test_empty_book_views_are_none():
ob = OrderBook()
assert ob.best_bid() is None
assert ob.best_ask() is None
assert ob.microprice() is None
assert ob.is_empty
v = ob.view()
assert v.mid is None
assert v.spread is None
assert v.imbalance == 0.0
def test_microprice_pulls_toward_thin_side():
ob = OrderBook(tick_size=0.01)
# bid side much heavier than ask side -> microprice near the ask
ob.apply_snapshot(bids=[(0.40, 1000)], asks=[(0.42, 10)], ts=1.0)
mp = ob.microprice(levels=1)
assert mp is not None
assert 0.41 < mp <= 0.42 # dragged up toward the thin ask
# symmetric sizes -> mid
ob.apply_snapshot(bids=[(0.40, 100)], asks=[(0.42, 100)], ts=2.0)
assert ob.microprice(levels=1) == pytest.approx(0.41)
def test_best_with_min_size_skips_dust():
ob = OrderBook(tick_size=0.01)
# a dust order (size 1) sits at the touch; real size is one level back
ob.apply_snapshot(bids=[(0.42, 1), (0.41, 500)], asks=[(0.44, 1), (0.45, 500)], ts=1.0)
price, size, top = ob.best_with_min_size(Side.BUY, min_size=5)
assert price == 0.41 and size == 500
assert top == 0.42 # the dust touch is still reported as top
price, size, top = ob.best_with_min_size(Side.SELL, min_size=5)
assert price == 0.45 and size == 500
assert top == 0.44
def test_depth_within_band():
ob = make_book()
# bids at 0.40,0.41,0.42 all within [0.40, 0.42]
assert ob.depth_within(Side.BUY, 0.40, 0.42) == 350
assert ob.depth_within(Side.BUY, 0.415, 0.42) == 50
def test_view_second_levels_and_imbalance():
ob = make_book()
v = ob.view(min_size=0.0)
assert v.best_bid == 0.42
assert v.second_bid == 0.41
assert v.best_ask == 0.44
assert v.second_ask == 0.45
assert -1.0 <= v.imbalance <= 1.0
def test_no_price_mirror():
assert to_no_price(0.42) == pytest.approx(0.58)
assert to_no_price(0.0) == 1.0
assert to_no_price(1.0) == 0.0
tests/test_quoting.py
+187
View File
@@ -0,0 +1,187 @@
"""Unit tests for pure quote construction — the strategy's decision core."""
from __future__ import annotations
import pytest
from polymaker.domain import Position, Regime, Side
from polymaker.strategy.quoting import (
QuoteInputs,
compute_fair_value,
construct_quotes,
round_to_tick,
)
from tests.conftest import view
def _inputs(meta, profile, **over):
base = dict(
meta=meta,
regime=Regime.QUIET,
fv=0.50,
vol_short=0.0,
toxicity=0.0,
yes_view=view(0.49, 0.51),
no_view=view(0.49, 0.51),
pos_yes=Position("yes-token"),
pos_no=Position("no-token"),
profile=profile,
now=1000.0,
)
base.update(over)
return QuoteInputs(**base)
# ── round_to_tick ──────────────────────────────────────────────────────────
def test_round_to_tick_down_and_up():
assert round_to_tick(0.5049, 0.01, 2, up=False) == 0.50
assert round_to_tick(0.5051, 0.01, 2, up=True) == 0.51
# clamps inside (0,1)
assert round_to_tick(0.0, 0.01, 2, up=False) == 0.01
assert round_to_tick(1.0, 0.01, 2, up=True) == 0.99
def test_compute_fair_value_flow_nudge():
# positive flow nudges FV up, negative down, no flow = microprice
assert compute_fair_value(0.50, 0.0, 0.01) == pytest.approx(0.50)
assert compute_fair_value(0.50, 1.0, 0.01, weight=0.5) == pytest.approx(0.505)
assert compute_fair_value(0.50, -1.0, 0.01, weight=0.5) == pytest.approx(0.495)
# ── two-sided quoting ────────────────────────────────────────────────────────
def test_quiet_market_quotes_both_sides_as_bids(meta, profile):
tq = construct_quotes(_inputs(meta, profile))
assert tq.regime == Regime.QUIET
yes = [q for q in tq.quotes if q.token_id == "yes-token"]
no = [q for q in tq.quotes if q.token_id == "no-token"]
assert yes and no
# both entry quotes are BUYs (USDC-collateralized two-sided quote)
assert all(q.side == Side.BUY for q in yes)
assert all(q.side == Side.BUY for q in no)
def test_pair_prices_sum_below_one(meta, profile):
"""BUY YES @ p and BUY NO @ q must satisfy p + q < 1 (merge edge)."""
tq = construct_quotes(_inputs(meta, profile))
top_yes = max(q.price for q in tq.quotes if q.token_id == "yes-token")
top_no = max(q.price for q in tq.quotes if q.token_id == "no-token")
assert top_yes + top_no < 1.0
def test_never_bids_through_fair_value(meta, profile):
"""No BUY should ever sit at or above FV - min_edge (YES) / (1-FV)-min_edge (NO)."""
tq = construct_quotes(_inputs(meta, profile, fv=0.50))
edge = profile.min_edge_ticks * meta.tick_size
for q in tq.quotes:
if q.side == Side.BUY and q.token_id == "yes-token":
assert q.price <= 0.50 - edge + 1e-9
if q.side == Side.BUY and q.token_id == "no-token":
assert q.price <= 0.50 - edge + 1e-9 # NO fv is also 0.50 here
def test_layers_split_size(meta, profile):
tq = construct_quotes(_inputs(meta, profile))
yes = sorted((q for q in tq.quotes if q.token_id == "yes-token" and q.side == Side.BUY),
key=lambda q: -q.price)
assert len(yes) == profile.layers
# deeper layer is at a lower price
assert yes[0].price > yes[1].price
# ── inventory skew ──────────────────────────────────────────────────────────
def test_long_yes_inventory_skews_quotes_down(meta, profile):
"""Holding YES should lower the YES bid and raise the NO bid vs flat."""
flat = construct_quotes(_inputs(meta, profile, vol_short=0.02))
longy = construct_quotes(
_inputs(meta, profile, vol_short=0.02, pos_yes=Position("yes-token", 300, 0.5))
)
def top(tq, tok):
ps = [q.price for q in tq.quotes if q.token_id == tok and q.side == Side.BUY]
return max(ps) if ps else None
# YES bid should not be higher when long YES; NO bid should not be lower
assert top(longy, "yes-token") <= top(flat, "yes-token")
assert top(longy, "no-token") >= top(flat, "no-token")
def test_reduce_only_emits_only_exits(meta, profile):
tq = construct_quotes(
_inputs(
meta, profile, regime=Regime.REDUCE_ONLY,
pos_yes=Position("yes-token", 100, 0.5),
)
)
assert all(q.side == Side.SELL for q in tq.quotes)
assert any(q.token_id == "yes-token" for q in tq.quotes)
def test_event_and_halted_pull_all_quotes(meta, profile):
for regime in (Regime.EVENT, Regime.HALTED):
tq = construct_quotes(
_inputs(meta, profile, regime=regime, pos_yes=Position("yes-token", 100, 0.5))
)
assert tq.is_empty
# ── exits ────────────────────────────────────────────────────────────────────
def test_exit_sell_priced_above_fv_when_not_urgent(meta, profile):
tq = construct_quotes(
_inputs(meta, profile, pos_yes=Position("yes-token", 100, 0.4), yes_exit_urgency=0.0)
)
sells = [q for q in tq.quotes if q.side == Side.SELL and q.token_id == "yes-token"]
assert sells
assert sells[0].price >= 0.50 # at/above FV, a passive maker exit
def test_exit_never_below_best_bid(meta, profile):
tq = construct_quotes(
_inputs(
meta, profile,
pos_yes=Position("yes-token", 100, 0.4),
yes_view=view(0.49, 0.51),
yes_exit_urgency=1.0, # maximally urgent
)
)
sells = [q for q in tq.quotes if q.side == Side.SELL and q.token_id == "yes-token"]
assert sells
assert sells[0].price >= 0.49 # still a maker order, never crosses down
def test_no_exit_when_position_is_dust(meta, profile):
tq = construct_quotes(
_inputs(meta, profile, pos_yes=Position("yes-token", 1.0, 0.4)) # below min_order_size
)
assert not [q for q in tq.quotes if q.side == Side.SELL]
# ── spread widening ──────────────────────────────────────────────────────────
def test_toxicity_widens_spread(meta, profile):
"""Higher toxicity should push the YES bid lower (wider spread)."""
calm = construct_quotes(_inputs(meta, profile, regime=Regime.TRENDING, toxicity=0.0))
toxic = construct_quotes(_inputs(meta, profile, regime=Regime.TRENDING, toxicity=0.02))
def top_yes(tq):
ps = [q.price for q in tq.quotes if q.token_id == "yes-token" and q.side == Side.BUY]
return max(ps) if ps else None
assert top_yes(toxic) < top_yes(calm)
def test_quiet_regime_clamps_spread_to_reward_band(meta, profile):
"""In QUIET, even with high vol the bid stays within the reward band of FV."""
tq = construct_quotes(_inputs(meta, profile, regime=Regime.QUIET, vol_short=0.5))
band = meta.rewards_max_spread / 100.0 # 0.03
top_yes = max(q.price for q in tq.quotes if q.token_id == "yes-token" and q.side == Side.BUY)
# bid should be within (band + a tick of rounding) of FV
assert top_yes >= 0.50 - band - meta.tick_size
tests/test_regime.py
+74
View File
@@ -0,0 +1,74 @@
"""Unit tests for the regime state machine."""
from __future__ import annotations
from polymaker.config import StrategyProfile
from polymaker.domain import Regime
from polymaker.strategy.regime import RegimeInputs, RegimeMachine
def _inp(**over):
base = dict(
now=1000.0,
tick=0.01,
fv=0.50,
prev_fv=0.50,
vol_ratio=1.0,
flow_z=0.0,
inventory_util=0.0,
hours_to_end=1000.0,
)
base.update(over)
return RegimeInputs(**base)
def test_default_is_quiet():
p = StrategyProfile()
assert RegimeMachine().decide(_inp(), p) == Regime.QUIET
def test_halts_take_priority():
p = StrategyProfile()
m = RegimeMachine()
assert m.decide(_inp(risk_halt=True), p) == Regime.HALTED
assert m.decide(_inp(ws_stale=True), p) == Regime.HALTED
assert m.decide(_inp(market_resolved=True), p) == Regime.HALTED
assert m.decide(_inp(hours_to_end=1.0), p) == Regime.HALTED # inside halt window
def test_fv_jump_triggers_event_and_cooloff():
p = StrategyProfile() # event_jump_ticks=8, cooloff=60
m = RegimeMachine()
# jump of 0.10 = 10 ticks > 8 -> EVENT
assert m.decide(_inp(fv=0.60, prev_fv=0.50), p) == Regime.EVENT
# still in cooloff a moment later even without a jump
assert m.decide(_inp(now=1030.0, fv=0.60, prev_fv=0.60), p) == Regime.EVENT
# after cooloff expires, back to quiet
assert m.decide(_inp(now=1100.0, fv=0.60, prev_fv=0.60), p) == Regime.QUIET
def test_sweep_flag_triggers_event():
p = StrategyProfile()
assert RegimeMachine().decide(_inp(sweep_flagged=True), p) == Regime.EVENT
def test_reduce_only_from_inventory_and_enddate():
p = StrategyProfile() # reduce_only_hours=24
m = RegimeMachine()
assert m.decide(_inp(inventory_util=1.0), p) == Regime.REDUCE_ONLY
assert m.decide(_inp(risk_reduce_only=True), p) == Regime.REDUCE_ONLY
assert m.decide(_inp(hours_to_end=12.0), p) == Regime.REDUCE_ONLY
def test_trending_from_flow_and_vol():
p = StrategyProfile() # trend_flow_z=1.5
m = RegimeMachine()
assert m.decide(_inp(flow_z=2.0), p) == Regime.TRENDING
assert m.decide(_inp(vol_ratio=3.0), p) == Regime.TRENDING
def test_event_beats_reduce_only_and_trending():
p = StrategyProfile()
m = RegimeMachine()
r = m.decide(_inp(sweep_flagged=True, inventory_util=1.0, flow_z=5.0), p)
assert r == Regime.EVENT
tests/test_risk.py
+86
View File
@@ -0,0 +1,86 @@
"""Tests for the RiskManager gates and circuit breakers."""
from __future__ import annotations
from polymaker.config import RiskConfig
from polymaker.domain import Fill, Side
from polymaker.risk.manager import RiskManager
from polymaker.state.store import StateStore
def _rm(tmp_path, **over):
cfg = RiskConfig(**{
"max_total_exposure_usdc": 5000, "max_market_notional_usdc": 800,
"max_event_group_loss_usdc": 1000, "daily_loss_kill_usdc": 250,
**over,
})
store = StateStore(tmp_path / "s.db")
return RiskManager(cfg, store), store
def test_daily_loss_kill_switch(tmp_path, meta):
rm, store = _rm(tmp_path)
# buy 1000 shares @ 0.50 -> -500 cash, +1000 inventory
store.apply_fill(Fill(meta.yes.token_id, Side.BUY, 0.50, 1000, "t1"))
rm.note_fill(Fill(meta.yes.token_id, Side.BUY, 0.50, 1000, "t1"))
rm.update_mark(meta.yes.token_id, 0.50)
rm.reset_day()
assert rm.global_halt()[0] is False
# fair value collapses to 0.20 -> unrealized loss 300 > 250 kill
rm.update_mark(meta.yes.token_id, 0.20)
halted, why = rm.global_halt()
assert halted and "daily_loss" in why
store.close()
def test_market_cap_triggers_reduce_only(tmp_path, meta):
rm, store = _rm(tmp_path, max_market_notional_usdc=100)
store.apply_fill(Fill(meta.yes.token_id, Side.BUY, 0.50, 300, "t1")) # 150 notional > 100
rm.update_mark(meta.yes.token_id, 0.50)
rm.update_mark(meta.no.token_id, 0.50)
d = rm.evaluate(meta, ws_stale=False, event_group_cost=0.0)
assert d.reduce_only and d.reason == "market_cap"
store.close()
def test_ws_stale_halts_market(tmp_path, meta):
rm, store = _rm(tmp_path)
d = rm.evaluate(meta, ws_stale=True, event_group_cost=0.0)
assert d.halt and d.reason == "ws_stale"
store.close()
def test_size_scale_tapers_near_cap(tmp_path, meta):
rm, store = _rm(tmp_path, max_market_notional_usdc=100)
# 85 notional -> 85% of cap -> should scale below 1.0 but not reduce-only
store.apply_fill(Fill(meta.yes.token_id, Side.BUY, 0.50, 170, "t1")) # 85 notional
rm.update_mark(meta.yes.token_id, 0.50)
rm.update_mark(meta.no.token_id, 0.50)
d = rm.evaluate(meta, ws_stale=False, event_group_cost=0.0)
assert not d.reduce_only
assert 0.0 < d.size_scale < 1.0
store.close()
def test_event_group_cap(tmp_path, meta):
rm, store = _rm(tmp_path, max_event_group_loss_usdc=50)
d = rm.evaluate(meta, ws_stale=False, event_group_cost=60.0)
assert d.reduce_only and d.reason == "event_group_cap"
store.close()
def test_error_rate_breaker(tmp_path, meta):
rm, store = _rm(tmp_path, max_order_error_rate=0.25)
for _ in range(15):
rm.note_order_result(False)
for _ in range(10):
rm.note_order_result(True) # 15/25 = 0.6 > 0.25
assert rm.global_halt()[0] is True
store.close()
def test_manual_kill(tmp_path, meta):
rm, store = _rm(tmp_path)
rm.kill()
assert rm.global_halt() == (True, "manual_kill")
store.close()
tests/test_state.py
+171
View File
@@ -0,0 +1,171 @@
"""Tests for StateStore, the user-event tracker, and the reconciler."""
from __future__ import annotations
from polymaker.domain import (
Fill,
OpenOrder,
OrderState,
Quote,
Regime,
Side,
TargetQuotes,
TradeState,
)
from polymaker.execution.reconciler import reconcile
from polymaker.state.store import StateStore
from polymaker.state.tracker import OrderEvent, TradeEvent, UserEventProcessor
# ── StateStore ──────────────────────────────────────────────────────────────
def test_apply_fill_updates_size_and_avg(tmp_path):
s = StateStore(tmp_path / "s.db")
s.apply_fill(Fill("tok", Side.BUY, 0.50, 100, "t1"))
assert s.position("tok").size == 100
assert s.position("tok").avg_price == 0.50
# buy more at a higher price -> weighted avg
s.apply_fill(Fill("tok", Side.BUY, 0.60, 100, "t2"))
assert s.position("tok").size == 200
assert abs(s.position("tok").avg_price - 0.55) < 1e-9
# sell reduces size, avg unchanged
s.apply_fill(Fill("tok", Side.SELL, 0.70, 50, "t3"))
assert s.position("tok").size == 150
assert abs(s.position("tok").avg_price - 0.55) < 1e-9
s.close()
def test_sell_to_flat_resets_avg(tmp_path):
s = StateStore(tmp_path / "s.db")
s.apply_fill(Fill("tok", Side.BUY, 0.5, 100, "t1"))
s.apply_fill(Fill("tok", Side.SELL, 0.6, 100, "t2"))
assert s.position("tok").size == 0
assert s.position("tok").avg_price == 0.0
s.close()
def test_reconcile_positions_skips_inflight_and_recent(tmp_path):
s = StateStore(tmp_path / "s.db")
s.mark_inflight("tok")
s.reconcile_positions({"tok": (999.0, 0.9)}) # ignored: in-flight
assert s.position("tok").size == 0
s.clear_inflight("tok")
# still recent fill guard: simulate no recent fill by using a fresh token
s.reconcile_positions({"other": (42.0, 0.3)})
assert s.position("other").size == 42.0
s.close()
def test_state_persists_across_restart(tmp_path):
db = tmp_path / "s.db"
s = StateStore(db)
s.apply_fill(Fill("tok", Side.BUY, 0.5, 100, "t1"))
s.close()
s2 = StateStore(db)
assert s2.position("tok").size == 100
s2.close()
# ── UserEventProcessor ───────────────────────────────────────────────────────
def test_matched_then_confirmed(tmp_path):
s = StateStore(tmp_path / "s.db")
changed: list[str] = []
p = UserEventProcessor(s, on_change=changed.append)
p.on_trade(TradeEvent("tok", Side.BUY, 0.5, 100, "trade1", TradeState.MATCHED, 1.0), "cid")
assert s.position("tok").size == 100
assert s.inflight("tok") == 1
p.on_trade(TradeEvent("tok", Side.BUY, 0.5, 100, "trade1", TradeState.CONFIRMED, 2.0), "cid")
assert s.inflight("tok") == 0
assert s.position("tok").size == 100 # settled
assert changed == ["cid", "cid"]
s.close()
def test_matched_is_idempotent(tmp_path):
s = StateStore(tmp_path / "s.db")
p = UserEventProcessor(s)
ev = TradeEvent("tok", Side.BUY, 0.5, 100, "trade1", TradeState.MATCHED, 1.0)
p.on_trade(ev, "cid")
p.on_trade(ev, "cid") # duplicate MATCHED for same trade id
assert s.position("tok").size == 100 # not doubled
s.close()
def test_failed_trade_reverses_fill(tmp_path):
s = StateStore(tmp_path / "s.db")
p = UserEventProcessor(s)
p.on_trade(TradeEvent("tok", Side.BUY, 0.5, 100, "trade1", TradeState.MATCHED, 1.0), "cid")
assert s.position("tok").size == 100
p.on_trade(TradeEvent("tok", Side.BUY, 0.5, 100, "trade1", TradeState.FAILED, 2.0), "cid")
assert s.position("tok").size == 0 # rolled back
assert s.inflight("tok") == 0
s.close()
def test_order_event_upsert_and_cancel(tmp_path):
s = StateStore(tmp_path / "s.db")
p = UserEventProcessor(s)
p.on_order(OrderEvent("o1", "tok", Side.BUY, 0.49, 100), "cid")
assert len(s.orders_for("tok")) == 1
p.on_order(OrderEvent("o1", "tok", Side.BUY, 0.49, 0, is_cancel=True), "cid")
assert len(s.orders_for("tok")) == 0
s.close()
# ── reconciler ───────────────────────────────────────────────────────────────
def _live(order_id, token, side, price, size):
return OpenOrder(order_id, token, side, price, size, OrderState.LIVE)
def test_reconcile_places_when_no_live():
tq = TargetQuotes("cid", Regime.QUIET, (Quote("tok", Side.BUY, 0.49, 100),))
plan = reconcile(tq, [], tick=0.01, reprice_ticks=2, resize_frac=0.15)
assert len(plan.to_place) == 1
assert plan.to_cancel == []
def test_reconcile_keeps_close_order():
tq = TargetQuotes("cid", Regime.QUIET, (Quote("tok", Side.BUY, 0.49, 100),))
live = [_live("o1", "tok", Side.BUY, 0.49, 102)] # within tolerances
plan = reconcile(tq, live, tick=0.01, reprice_ticks=2, resize_frac=0.15)
assert plan.is_noop
def test_reconcile_reprices_when_far():
tq = TargetQuotes("cid", Regime.QUIET, (Quote("tok", Side.BUY, 0.45, 100),))
live = [_live("o1", "tok", Side.BUY, 0.49, 100)] # 4 ticks away > 2
plan = reconcile(tq, live, tick=0.01, reprice_ticks=2, resize_frac=0.15)
assert plan.to_cancel == ["o1"]
assert len(plan.to_place) == 1
def test_reconcile_resizes_when_size_drifts():
tq = TargetQuotes("cid", Regime.QUIET, (Quote("tok", Side.BUY, 0.49, 100),))
live = [_live("o1", "tok", Side.BUY, 0.49, 50)] # 50% smaller > 15%
plan = reconcile(tq, live, tick=0.01, reprice_ticks=2, resize_frac=0.15)
assert plan.to_cancel == ["o1"]
assert len(plan.to_place) == 1
def test_reconcile_cancels_all_when_target_empty():
tq = TargetQuotes("cid", Regime.EVENT, ())
live = [_live("o1", "tok", Side.BUY, 0.49, 100), _live("o2", "tok", Side.SELL, 0.55, 50)]
plan = reconcile(tq, live, tick=0.01, reprice_ticks=2, resize_frac=0.15)
assert set(plan.to_cancel) == {"o1", "o2"}
assert plan.to_place == []
def test_reconcile_matches_layers_one_to_one():
tq = TargetQuotes("cid", Regime.QUIET, (
Quote("tok", Side.BUY, 0.49, 100),
Quote("tok", Side.BUY, 0.47, 100),
))
live = [_live("o1", "tok", Side.BUY, 0.49, 100)] # only the top layer exists
plan = reconcile(tq, live, tick=0.01, reprice_ticks=2, resize_frac=0.15)
assert plan.to_cancel == []
assert len(plan.to_place) == 1 # only the missing deeper layer
assert plan.to_place[0].price == 0.47
tests/test_userstream_parse.py
+88
View File
@@ -0,0 +1,88 @@
"""Tests for user-WS normalization (maker fill extraction + order tracking).
Frames modeled on v1's observed shape; reconfirm field names in the wallet spike.
"""
from __future__ import annotations
from polymaker.domain import Side, TradeState
from polymaker.userstream.parse import normalize_order, normalize_trade
OUR = "0xMyWallet"
def _other(token: str) -> str | None:
return {"yes-tok": "no-tok", "no-tok": "yes-tok"}.get(token)
def test_maker_same_outcome_is_a_sell():
# taker BUYs YES; we are the maker on YES -> we SELL YES
msg = {
"event_type": "trade",
"market": "0xcond",
"asset_id": "yes-tok",
"side": "BUY",
"outcome": "Yes",
"status": "MATCHED",
"id": "trade1",
"timestamp": "1700000000000",
"maker_orders": [
{"maker_address": OUR, "matched_amount": "50", "price": "0.49", "outcome": "Yes"}
],
}
evs = normalize_trade(msg, OUR, _other)
assert len(evs) == 1
ev = evs[0]
assert ev.token_id == "yes-tok"
assert ev.our_side is Side.SELL
assert ev.size == 50 and ev.price == 0.49
assert ev.status is TradeState.MATCHED
def test_maker_different_outcome_is_a_mint_buy():
# taker BUYs YES; we are the maker on NO -> a mint: we BUY NO
msg = {
"event_type": "trade",
"market": "0xcond",
"asset_id": "yes-tok",
"side": "BUY",
"outcome": "Yes",
"status": "MATCHED",
"id": "trade2",
"timestamp": "1700000000000",
"maker_orders": [
{"maker_address": OUR, "matched_amount": "30", "price": "0.51", "outcome": "No"}
],
}
evs = normalize_trade(msg, OUR, _other)
assert len(evs) == 1
assert evs[0].token_id == "no-tok"
assert evs[0].our_side is Side.BUY
assert evs[0].size == 30
def test_ignores_maker_orders_that_are_not_ours():
msg = {
"event_type": "trade", "market": "0xcond", "asset_id": "yes-tok", "side": "BUY",
"outcome": "Yes", "status": "MATCHED", "id": "t", "timestamp": "1700000000000",
"maker_orders": [
{"maker_address": "0xSomeoneElse", "matched_amount": "50", "price": "0.49", "outcome": "Yes"}
],
}
assert normalize_trade(msg, OUR, _other) == []
def test_normalize_order_remaining_and_cancel():
ev = normalize_order({
"event_type": "order", "asset_id": "yes-tok", "side": "BUY", "price": "0.49",
"original_size": "100", "size_matched": "40", "status": "LIVE", "id": "o1",
})
assert ev is not None
assert ev.remaining_size == 60
assert ev.is_cancel is False
cancel = normalize_order({
"event_type": "order", "asset_id": "yes-tok", "side": "BUY", "price": "0.49",
"original_size": "100", "size_matched": "0", "status": "CANCELED", "id": "o1",
})
assert cancel is not None and cancel.is_cancel is True
trading.py
-471
View File
@@ -1,471 +0,0 @@
import gc # Garbage collection
import os # Operating system interface
import json # JSON handling
import asyncio # Asynchronous I/O
import traceback # Exception handling
import pandas as pd # Data analysis library
import math # Mathematical functions
import poly_data.global_state as global_state
import poly_data.CONSTANTS as CONSTANTS
# Import utility functions for trading
from poly_data.trading_utils import get_best_bid_ask_deets, get_order_prices, get_buy_sell_amount, round_down, round_up
from poly_data.data_utils import get_position, get_order, set_position
# Create directory for storing position risk information
if not os.path.exists('positions/'):
os.makedirs('positions/')
def send_buy_order(order):
"""
Create a BUY order for a specific token.
This function:
1. Cancels any existing orders for the token
2. Checks if the order price is within acceptable range
3. Creates a new buy order if conditions are met
Args:
order (dict): Order details including token, price, size, and market parameters
"""
client = global_state.client
# Only cancel existing orders if we need to make significant changes
existing_buy_size = order['orders']['buy']['size']
existing_buy_price = order['orders']['buy']['price']
# Cancel orders if price changed significantly or size needs major adjustment
price_diff = abs(existing_buy_price - order['price']) if existing_buy_price > 0 else float('inf')
size_diff = abs(existing_buy_size - order['size']) if existing_buy_size > 0 else float('inf')
should_cancel = (
price_diff > 0.005 or # Cancel if price diff > 0.5 cents
size_diff > order['size'] * 0.1 or # Cancel if size diff > 10%
existing_buy_size == 0 # Cancel if no existing buy order
)
if should_cancel and (existing_buy_size > 0 or order['orders']['sell']['size'] > 0):
print(f"Cancelling buy orders - price diff: {price_diff:.4f}, size diff: {size_diff:.1f}")
client.cancel_all_asset(order['token'])
elif not should_cancel:
print(f"Keeping existing buy orders - minor changes: price diff: {price_diff:.4f}, size diff: {size_diff:.1f}")
return # Don't place new order if existing one is fine
# Calculate minimum acceptable price based on market spread
incentive_start = order['mid_price'] - order['max_spread']/100
trade = True
# Don't place orders that are below incentive threshold
if order['price'] < incentive_start:
trade = False
if trade:
# Only place orders with prices between 0.1 and 0.9 to avoid extreme positions
if order['price'] >= 0.1 and order['price'] < 0.9:
print(f'Creating new order for {order["size"]} at {order["price"]}')
print(order['token'], 'BUY', order['price'], order['size'])
client.create_order(
order['token'],
'BUY',
order['price'],
order['size'],
True if order['neg_risk'] == 'TRUE' else False
)
else:
print("Not creating buy order because its outside acceptable price range (0.1-0.9)")
else:
print(f'Not creating new order because order price of {order["price"]} is less than incentive start price of {incentive_start}. Mid price is {order["mid_price"]}')
def send_sell_order(order):
"""
Create a SELL order for a specific token.
This function:
1. Cancels any existing orders for the token
2. Creates a new sell order with the specified parameters
Args:
order (dict): Order details including token, price, size, and market parameters
"""
client = global_state.client
# Only cancel existing orders if we need to make significant changes
existing_sell_size = order['orders']['sell']['size']
existing_sell_price = order['orders']['sell']['price']
# Cancel orders if price changed significantly or size needs major adjustment
price_diff = abs(existing_sell_price - order['price']) if existing_sell_price > 0 else float('inf')
size_diff = abs(existing_sell_size - order['size']) if existing_sell_size > 0 else float('inf')
should_cancel = (
price_diff > 0.005 or # Cancel if price diff > 0.5 cents
size_diff > order['size'] * 0.1 or # Cancel if size diff > 10%
existing_sell_size == 0 # Cancel if no existing sell order
)
if should_cancel and (existing_sell_size > 0 or order['orders']['buy']['size'] > 0):
print(f"Cancelling sell orders - price diff: {price_diff:.4f}, size diff: {size_diff:.1f}")
client.cancel_all_asset(order['token'])
elif not should_cancel:
print(f"Keeping existing sell orders - minor changes: price diff: {price_diff:.4f}, size diff: {size_diff:.1f}")
return # Don't place new order if existing one is fine
print(f'Creating new order for {order["size"]} at {order["price"]}')
client.create_order(
order['token'],
'SELL',
order['price'],
order['size'],
True if order['neg_risk'] == 'TRUE' else False
)
# Dictionary to store locks for each market to prevent concurrent trading on the same market
market_locks = {}
async def perform_trade(market):
"""
Main trading function that handles market making for a specific market.
This function:
1. Merges positions when possible to free up capital
2. Analyzes the market to determine optimal bid/ask prices
3. Manages buy and sell orders based on position size and market conditions
4. Implements risk management with stop-loss and take-profit logic
Args:
market (str): The market ID to trade on
"""
# Create a lock for this market if it doesn't exist
if market not in market_locks:
market_locks[market] = asyncio.Lock()
# Use lock to prevent concurrent trading on the same market
async with market_locks[market]:
try:
client = global_state.client
# Get market details from the configuration
row = global_state.df[global_state.df['condition_id'] == market].iloc[0]
# Determine decimal precision from tick size
round_length = len(str(row['tick_size']).split(".")[1])
# Get trading parameters for this market type
params = global_state.params[row['param_type']]
# Create a list with both outcomes for the market
deets = [
{'name': 'token1', 'token': row['token1'], 'answer': row['answer1']},
{'name': 'token2', 'token': row['token2'], 'answer': row['answer2']}
]
print(f"\n\n{pd.Timestamp.utcnow().tz_localize(None)}: {row['question']}")
# Get current positions for both outcomes
pos_1 = get_position(row['token1'])['size']
pos_2 = get_position(row['token2'])['size']
# ------- POSITION MERGING LOGIC -------
# Calculate if we have opposing positions that can be merged
amount_to_merge = min(pos_1, pos_2)
# Only merge if positions are above minimum threshold
if float(amount_to_merge) > CONSTANTS.MIN_MERGE_SIZE:
# Get exact position sizes from blockchain for merging
pos_1 = client.get_position(row['token1'])[0]
pos_2 = client.get_position(row['token2'])[0]
amount_to_merge = min(pos_1, pos_2)
scaled_amt = amount_to_merge / 10**6
if scaled_amt > CONSTANTS.MIN_MERGE_SIZE:
print(f"Position 1 is of size {pos_1} and Position 2 is of size {pos_2}. Merging positions")
# Execute the merge operation
client.merge_positions(amount_to_merge, market, row['neg_risk'] == 'TRUE')
# Update our local position tracking
set_position(row['token1'], 'SELL', scaled_amt, 0, 'merge')
set_position(row['token2'], 'SELL', scaled_amt, 0, 'merge')
# ------- TRADING LOGIC FOR EACH OUTCOME -------
# Loop through both outcomes in the market (YES and NO)
for detail in deets:
token = int(detail['token'])
# Get current orders for this token
orders = get_order(token)
# Get market depth and price information
deets = get_best_bid_ask_deets(market, detail['name'], 100, 0.1)
#if deet has None for one these values below, call it with min size of 20
if deets['best_bid'] is None or deets['best_ask'] is None or deets['best_bid_size'] is None or deets['best_ask_size'] is None:
deets = get_best_bid_ask_deets(market, detail['name'], 20, 0.1)
# Extract all order book details
best_bid = deets['best_bid']
best_bid_size = deets['best_bid_size']
second_best_bid = deets['second_best_bid']
second_best_bid_size = deets['second_best_bid_size']
top_bid = deets['top_bid']
best_ask = deets['best_ask']
best_ask_size = deets['best_ask_size']
second_best_ask = deets['second_best_ask']
second_best_ask_size = deets['second_best_ask_size']
top_ask = deets['top_ask']
# Round prices to appropriate precision
best_bid = round(best_bid, round_length)
best_ask = round(best_ask, round_length)
# Calculate ratio of buy vs sell liquidity in the market
try:
overall_ratio = (deets['bid_sum_within_n_percent']) / (deets['ask_sum_within_n_percent'])
except:
overall_ratio = 0
try:
second_best_bid = round(second_best_bid, round_length)
second_best_ask = round(second_best_ask, round_length)
except:
pass
top_bid = round(top_bid, round_length)
top_ask = round(top_ask, round_length)
# Get our current position and average price
pos = get_position(token)
position = pos['size']
avgPrice = pos['avgPrice']
position = round_down(position, 2)
# Calculate optimal bid and ask prices based on market conditions
bid_price, ask_price = get_order_prices(
best_bid, best_bid_size, top_bid, best_ask,
best_ask_size, top_ask, avgPrice, row
)
bid_price = round(bid_price, round_length)
ask_price = round(ask_price, round_length)
# Calculate mid price for reference
mid_price = (top_bid + top_ask) / 2
# Log market conditions for this outcome
print(f"\nFor {detail['answer']}. Orders: {orders} Position: {position}, "
f"avgPrice: {avgPrice}, Best Bid: {best_bid}, Best Ask: {best_ask}, "
f"Bid Price: {bid_price}, Ask Price: {ask_price}, Mid Price: {mid_price}")
# Get position for the opposite token to calculate total exposure
other_token = global_state.REVERSE_TOKENS[str(token)]
other_position = get_position(other_token)['size']
# Calculate how much to buy or sell based on our position
buy_amount, sell_amount = get_buy_sell_amount(position, bid_price, row, other_position)
# Get max_size for logging (same logic as in get_buy_sell_amount)
max_size = row.get('max_size', row['trade_size'])
# Prepare order object with all necessary information
order = {
"token": token,
"mid_price": mid_price,
"neg_risk": row['neg_risk'],
"max_spread": row['max_spread'],
'orders': orders,
'token_name': detail['name'],
'row': row
}
print(f"Position: {position}, Other Position: {other_position}, "
f"Trade Size: {row['trade_size']}, Max Size: {max_size}, "
f"buy_amount: {buy_amount}, sell_amount: {sell_amount}")
# File to store risk management information for this market
fname = 'positions/' + str(market) + '.json'
# ------- SELL ORDER LOGIC -------
if sell_amount > 0:
# Skip if we have no average price (no real position)
if avgPrice == 0:
print("Avg Price is 0. Skipping")
continue
order['size'] = sell_amount
order['price'] = ask_price
# Get fresh market data for risk assessment
n_deets = get_best_bid_ask_deets(market, detail['name'], 100, 0.1)
# Calculate current market price and spread
mid_price = round_up((n_deets['best_bid'] + n_deets['best_ask']) / 2, round_length)
spread = round(n_deets['best_ask'] - n_deets['best_bid'], 2)
# Calculate current profit/loss on position
pnl = (mid_price - avgPrice) / avgPrice * 100
print(f"Mid Price: {mid_price}, Spread: {spread}, PnL: {pnl}")
# Prepare risk details for tracking
risk_details = {
'time': str(pd.Timestamp.utcnow().tz_localize(None)),
'question': row['question']
}
try:
ratio = (n_deets['bid_sum_within_n_percent']) / (n_deets['ask_sum_within_n_percent'])
except:
ratio = 0
pos_to_sell = sell_amount # Amount to sell in risk-off scenario
# ------- STOP-LOSS LOGIC -------
# Trigger stop-loss if either:
# 1. PnL is below threshold and spread is tight enough to exit
# 2. Volatility is too high
if (pnl < params['stop_loss_threshold'] and spread <= params['spread_threshold']) or row['3_hour'] > params['volatility_threshold']:
risk_details['msg'] = (f"Selling {pos_to_sell} because spread is {spread} and pnl is {pnl} "
f"and ratio is {ratio} and 3 hour volatility is {row['3_hour']}")
print("Stop loss Triggered: ", risk_details['msg'])
# Sell at market best bid to ensure execution
order['size'] = pos_to_sell
order['price'] = n_deets['best_bid']
# Set period to avoid trading after stop-loss
risk_details['sleep_till'] = str(pd.Timestamp.utcnow().tz_localize(None) +
pd.Timedelta(hours=params['sleep_period']))
print("Risking off")
send_sell_order(order)
client.cancel_all_market(market)
# Save risk details to file
open(fname, 'w').write(json.dumps(risk_details))
continue
# ------- BUY ORDER LOGIC -------
# Get max_size, defaulting to trade_size if not specified
max_size = row.get('max_size', row['trade_size'])
# Only buy if:
# 1. Position is less than max_size (new logic)
# 2. Position is less than absolute cap (250)
# 3. Buy amount is above minimum size
if position < max_size and position < 250 and buy_amount > 0 and buy_amount >= row['min_size']:
# Get reference price from market data
sheet_value = row['best_bid']
if detail['name'] == 'token2':
sheet_value = 1 - row['best_ask']
sheet_value = round(sheet_value, round_length)
order['size'] = buy_amount
order['price'] = bid_price
# Check if price is far from reference
price_change = abs(order['price'] - sheet_value)
send_buy = True
# ------- RISK-OFF PERIOD CHECK -------
# If we're in a risk-off period (after stop-loss), don't buy
if os.path.isfile(fname):
risk_details = json.load(open(fname))
start_trading_at = pd.to_datetime(risk_details['sleep_till'])
current_time = pd.Timestamp.utcnow().tz_localize(None)
print(risk_details, current_time, start_trading_at)
if current_time < start_trading_at:
send_buy = False
print(f"Not sending a buy order because recently risked off. "
f"Risked off at {risk_details['time']}")
# Only proceed if we're not in risk-off period
if send_buy:
# Don't buy if volatility is high or price is far from reference
if row['3_hour'] > params['volatility_threshold'] or price_change >= 0.05:
print(f'3 Hour Volatility of {row["3_hour"]} is greater than max volatility of '
f'{params["volatility_threshold"]} or price of {order["price"]} is outside '
f'0.05 of {sheet_value}. Cancelling all orders')
client.cancel_all_asset(order['token'])
else:
# Check for reverse position (holding opposite outcome)
rev_token = global_state.REVERSE_TOKENS[str(token)]
rev_pos = get_position(rev_token)
# If we have significant opposing position, don't buy more
if rev_pos['size'] > row['min_size']:
print("Bypassing creation of new buy order because there is a reverse position")
if orders['buy']['size'] > CONSTANTS.MIN_MERGE_SIZE:
print("Cancelling buy orders because there is a reverse position")
client.cancel_all_asset(order['token'])
continue
# Check market buy/sell volume ratio
if overall_ratio < 0:
send_buy = False
print(f"Not sending a buy order because overall ratio is {overall_ratio}")
client.cancel_all_asset(order['token'])
else:
# Place new buy order if any of these conditions are met:
# 1. We can get a better price than current order
if best_bid > orders['buy']['price']:
print(f"Sending Buy Order for {token} because better price. "
f"Orders look like this: {orders['buy']}. Best Bid: {best_bid}")
send_buy_order(order)
# 2. Current position + orders is not enough to reach max_size
elif position + orders['buy']['size'] < 0.95 * max_size:
print(f"Sending Buy Order for {token} because not enough position + size")
send_buy_order(order)
# 3. Our current order is too large and needs to be resized
elif orders['buy']['size'] > order['size'] * 1.01:
print(f"Resending buy orders because open orders are too large")
send_buy_order(order)
# Commented out logic for cancelling orders when market conditions change
# elif best_bid_size < orders['buy']['size'] * 0.98 and abs(best_bid - second_best_bid) > 0.03:
# print(f"Cancelling buy orders because best size is less than 90% of open orders and spread is too large")
# global_state.client.cancel_all_asset(order['token'])
# ------- TAKE PROFIT / SELL ORDER MANAGEMENT -------
elif sell_amount > 0:
order['size'] = sell_amount
# Calculate take-profit price based on average cost
tp_price = round_up(avgPrice + (avgPrice * params['take_profit_threshold']/100), round_length)
order['price'] = round_up(tp_price if ask_price < tp_price else ask_price, round_length)
tp_price = float(tp_price)
order_price = float(orders['sell']['price'])
# Calculate % difference between current order and ideal price
diff = abs(order_price - tp_price)/tp_price * 100
# Update sell order if:
# 1. Current order price is significantly different from target
if diff > 2:
print(f"Sending Sell Order for {token} because better current order price of "
f"{order_price} is deviant from the tp_price of {tp_price} and diff is {diff}")
send_sell_order(order)
# 2. Current order size is too small for our position
elif orders['sell']['size'] < position * 0.97:
print(f"Sending Sell Order for {token} because not enough sell size. "
f"Position: {position}, Sell Size: {orders['sell']['size']}")
send_sell_order(order)
# Commented out additional conditions for updating sell orders
# elif orders['sell']['price'] < ask_price:
# print(f"Updating Sell Order for {token} because its not at the right price")
# send_sell_order(order)
# elif best_ask_size < orders['sell']['size'] * 0.98 and abs(best_ask - second_best_ask) > 0.03...:
# print(f"Cancelling sell orders because best size is less than 90% of open orders...")
# send_sell_order(order)
except Exception as ex:
print(f"Error performing trade for {market}: {ex}")
traceback.print_exc()
# Clean up memory and introduce a small delay
gc.collect()
await asyncio.sleep(2)
update_markets.py
-133
View File
@@ -1,133 +0,0 @@
import time
import pandas as pd
from data_updater.trading_utils import get_clob_client
from data_updater.google_utils import get_spreadsheet
from data_updater.find_markets import get_sel_df, get_all_markets, get_all_results, get_markets, add_volatility_to_df
from gspread_dataframe import set_with_dataframe
import traceback
# Initialize global variables
spreadsheet = get_spreadsheet()
client = get_clob_client()
wk_all = spreadsheet.worksheet("All Markets")
wk_vol = spreadsheet.worksheet("Volatility Markets")
sel_df = get_sel_df(spreadsheet, "Selected Markets")
def update_sheet(data, worksheet):
all_values = worksheet.get_all_values()
existing_num_rows = len(all_values)
existing_num_cols = len(all_values[0]) if all_values else 0
num_rows, num_cols = data.shape
max_rows = max(num_rows, existing_num_rows)
max_cols = max(num_cols, existing_num_cols)
# Create a DataFrame with the maximum size and fill it with empty strings
padded_data = pd.DataFrame('', index=range(max_rows), columns=range(max_cols))
# Update the padded DataFrame with the original data and its columns
padded_data.iloc[:num_rows, :num_cols] = data.values
padded_data.columns = list(data.columns) + [''] * (max_cols - num_cols)
# Update the sheet with the padded DataFrame, including column headers
set_with_dataframe(worksheet, padded_data, include_index=False, include_column_header=True, resize=True)
def sort_df(df):
# Calculate the mean and standard deviation for each column
mean_gm = df['gm_reward_per_100'].mean()
std_gm = df['gm_reward_per_100'].std()
mean_volatility = df['volatility_sum'].mean()
std_volatility = df['volatility_sum'].std()
# Standardize the columns
df['std_gm_reward_per_100'] = (df['gm_reward_per_100'] - mean_gm) / std_gm
df['std_volatility_sum'] = (df['volatility_sum'] - mean_volatility) / std_volatility
# Define a custom scoring function for best_bid and best_ask
def proximity_score(value):
if 0.1 <= value <= 0.25:
return (0.25 - value) / 0.15
elif 0.75 <= value <= 0.9:
return (value - 0.75) / 0.15
else:
return 0
df['bid_score'] = df['best_bid'].apply(proximity_score)
df['ask_score'] = df['best_ask'].apply(proximity_score)
# Create a composite score (higher is better for rewards, lower is better for volatility, with proximity scores)
df['composite_score'] = (
df['std_gm_reward_per_100'] -
df['std_volatility_sum'] +
df['bid_score'] +
df['ask_score']
)
# Sort by the composite score in descending order
sorted_df = df.sort_values(by='composite_score', ascending=False)
# Drop the intermediate columns used for calculation
sorted_df = sorted_df.drop(columns=['std_gm_reward_per_100', 'std_volatility_sum', 'bid_score', 'ask_score', 'composite_score'])
return sorted_df
def fetch_and_process_data():
global spreadsheet, client, wk_all, wk_vol, sel_df
spreadsheet = get_spreadsheet()
client = get_clob_client()
wk_all = spreadsheet.worksheet("All Markets")
wk_vol = spreadsheet.worksheet("Volatility Markets")
wk_full = spreadsheet.worksheet("Full Markets")
sel_df = get_sel_df(spreadsheet, "Selected Markets")
all_df = get_all_markets(client)
print("Got all Markets")
all_results = get_all_results(all_df, client)
print("Got all Results")
m_data, all_markets = get_markets(all_results, sel_df, maker_reward=0.75)
print("Got all orderbook")
print(f'{pd.to_datetime("now")}: Fetched all markets data of length {len(all_markets)}.')
new_df = add_volatility_to_df(all_markets)
new_df['volatility_sum'] = new_df['24_hour'] + new_df['7_day'] + new_df['14_day']
new_df = new_df.sort_values('volatility_sum', ascending=True)
new_df['volatilty/reward'] = ((new_df['gm_reward_per_100'] / new_df['volatility_sum']).round(2)).astype(str)
new_df = new_df[['question', 'answer1', 'answer2', 'spread', 'rewards_daily_rate', 'gm_reward_per_100', 'sm_reward_per_100', 'bid_reward_per_100', 'ask_reward_per_100', 'volatility_sum', 'volatilty/reward', 'min_size', '1_hour', '3_hour', '6_hour', '12_hour', '24_hour', '7_day', '30_day',
'best_bid', 'best_ask', 'volatility_price', 'max_spread', 'tick_size',
'neg_risk', 'market_slug', 'token1', 'token2', 'condition_id']]
volatility_df = new_df.copy()
volatility_df = volatility_df[new_df['volatility_sum'] < 20]
# volatility_df = sort_df(volatility_df)
volatility_df = volatility_df.sort_values('gm_reward_per_100', ascending=False)
new_df = new_df.sort_values('gm_reward_per_100', ascending=False)
print(f'{pd.to_datetime("now")}: Fetched select market of length {len(new_df)}.')
if len(new_df) > 50:
update_sheet(new_df, wk_all)
update_sheet(volatility_df, wk_vol)
update_sheet(m_data, wk_full)
else:
print(f'{pd.to_datetime("now")}: Not updating sheet because of length {len(new_df)}.')
if __name__ == "__main__":
while True:
try:
fetch_and_process_data()
time.sleep(60 * 60) # Sleep for an hour
except Exception as e:
traceback.print_exc()
print(str(e))
update_stats.py
-18
View File
@@ -1,18 +0,0 @@
from poly_data.polymarket_client import PolymarketClient
from poly_stats.account_stats import update_stats_once
import pandas as pd
import time
import traceback
client = PolymarketClient()
if __name__ == '__main__':
while True:
try:
update_stats_once(client)
except Exception as e:
traceback.print_exc()
print("Now sleeping\n")
time.sleep(60 * 60 * 3) #3 hours
uv.lock
Generated
+1209 -2037
View File
File diff suppressed because it is too large Load Diff