Compare commits

...

30 Commits

Author SHA1 Message Date
schrodinger01 2f89945ce1 feat(ingestor): enrich market metadata with gamma-api volume/liquidity
The CLOB API does not expose 24h volume or order-book liquidity, so the
size_anomaly detector currently has no real ratio to compare a trade
against and falls back to the niche-base 0.2 confidence floor. That makes
the volume_impact / book_impact thresholds essentially dead code.

This change adds a small client for the public gamma-api markets endpoint
and merges its volume24hr / liquidityNum snapshot into MarketMetadata
during the existing periodic sync. The detector can now compute real
volume and book impact ratios.

Notes on the gamma client:
- gamma-api enforces a server-side max of 100 markets per page and caps
  `offset` around 10000. The client paginates with bounded concurrency,
  sorted by `volume24hr desc`, so the most-traded markets (which is where
  size anomalies actually matter) are always covered. Markets beyond that
  window have negligible recent volume and the niche path handles them
  fine without a ratio.
- Failures are swallowed: a degraded gamma endpoint must not stop CLOB
  metadata from being cached, since size_anomaly + the niche path remain
  functional with `daily_volume=None`.

MarketMetadata gains three optional Decimal fields (daily_volume,
weekly_volume, liquidity); to_dict / from_dict round-trip is preserved
and older cache entries without these keys deserialize cleanly.

Tests: 12 new tests for GammaClient (parsing, single-page, short-page
stop, offset-cap clean stop, retry, malformed responses); existing
metadata_sync tests updated to inject a mocked GammaClient so they don't
hit the real network.
2026-06-14 19:12:27 +00:00
Patrick Selamy b962bdaee2 fix(ingestor): align WebSocket subscribe + routing with live API (#105)
* fix(ingestor): align WebSocket subscribe + routing with live API

The Polymarket ws-live-data WebSocket requires `action: "subscribe"` in
the subscribe envelope. Without it the server accepts the connection but
never delivers trade events, causing the tracker to silently produce
zero alerts.

Additionally, incoming frames are shaped `{connection_id, payload:{...}}`
-- they do NOT echo the `topic`/`type` keys we sent. The previous routing
check matched nothing and every real trade was silently dropped.

Changes:
- Add `action: "subscribe"` to subscription message
- Route incoming messages by payload shape (transactionHash + proxyWallet)
- Add ratchet tests for payload routing edge cases
- Rewrite README as agent-first with <2min quickstart
- Add skill draft (docs/skill-tracking-prediction-market-flow.md)

Closes #89

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(lint): remove unused imports in test_pipeline_persistence

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style: apply ruff formatting

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-14 14:24:54 -04:00
axel-claw ff678468e3 fix: load nested env settings and ignore blank webhooks (#91) 2026-04-11 16:20:15 -04:00
Patrick Selamy 1f4fc607a7 feat: wire WalletRepository and FundingRepository into live pipeline (#90)
The live pipeline was receiving trade data and populating Redis caches but
never persisting wallet profiles or funding transfers to Postgres. This
wires the repositories into the _on_trade flow: when a fresh wallet signal
is detected, the wallet profile is upserted to wallet_profiles and the
funding chain is traced and inserted into funding_transfers. Existing
Redis caching behavior is preserved.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-01 09:21:37 -04:00
Patrick Selamy 232680a760 chore: smoke test fixes and code quality improvements (#88)
* fix: replace deprecated datetime.utcnow() and websockets.legacy APIs

- Replace datetime.utcnow() with datetime.now(UTC) in Orderbook model
- Use websockets.asyncio.client.connect instead of legacy websockets.connect
- Import ConnectionClosed from websockets.exceptions directly
- Update test mocks to patch the new import paths

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: resolve ruff lint errors in alembic migration

- Replace typing.Union with X | Y syntax (UP007)
- Import Sequence from collections.abc instead of typing (UP035)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: remove deprecated version key from docker-compose.yml

The top-level 'version' key is obsolete in modern Docker Compose
and produces a warning.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: fix README to match actual project structure and tooling

- Replace pip commands with uv equivalents
- Fix run command from 'python -m src.main' to 'python -m polymarket_insider_tracker'
- Update project structure tree to reflect actual src/polymarket_insider_tracker/ layout

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 22:03:10 -04:00
Patrick Selamy 7c494c38a6 fix: replace (str, Enum) with StrEnum to resolve ruff UP042 (#87)
StrEnum (Python 3.11+) is the modern replacement for the (str, Enum)
pattern. Updates SyncState and PipelineState enums.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 21:17:51 -04:00
Patrick Selamy 17504a5eaa fix: replace (str, Enum) with StrEnum to resolve ruff UP042 (#86)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 21:16:23 -04:00
Patrick Selamy cadc593d62 fix: replace (str, Enum) with StrEnum to resolve ruff UP042 (#85)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 21:14:36 -04:00
Patrick Selamy 0320a96f8a fix: replace (str, Enum) with StrEnum (ruff UP042) (#84)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 21:12:59 -04:00
Patrick Selamy 753d6cf1de fix: replace (str, Enum) with StrEnum (ruff UP042) (#83)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 21:11:19 -04:00
Patrick Selamy f3801987fe fix: resolve .env.example variable interpolation and alembic env var mismatch (#62) (#82)
.env files don't support shell variable expansion, so DATABASE_URL and
REDIS_URL contained literal ${...} strings. Also alembic/env.py read
SQLALCHEMY_DATABASE_URL instead of DATABASE_URL.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 20:05:00 -04:00
Patrick Selamy 5cbcaccd82 fix: resolve alembic migration failures from .env and env var mismatch (#81)
- Replace shell variable interpolation in .env.example with literal
  values since .env files don't support variable expansion
- Change alembic/env.py to read DATABASE_URL (matching app convention)
  instead of SQLALCHEMY_DATABASE_URL

Closes #62

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 20:03:58 -04:00
Patrick Selamy 288fbc75cf fix: resolve broken alembic migrations (#62) (#80)
- Replace shell variable interpolation in .env.example with literal values
- Change alembic/env.py to read DATABASE_URL instead of SQLALCHEMY_DATABASE_URL

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 20:02:53 -04:00
Patrick Selamy 37ea0c6540 fix: resolve broken alembic migrations (#62) (#79)
- Replace shell variable interpolation in .env.example with literal values
- Change alembic/env.py to read DATABASE_URL instead of SQLALCHEMY_DATABASE_URL

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 20:02:02 -04:00
Patrick Selamy 654f4466f0 fix: resolve broken alembic migrations (#62) (#78)
- Replace shell variable interpolation in .env.example with literal
  values since .env files don't expand variables
- Change alembic/env.py to read DATABASE_URL instead of
  SQLALCHEMY_DATABASE_URL to match the app's env var name

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 20:00:56 -04:00
Patrick Selamy 8628e3aadb fix: use literal values in .env.example and correct env var name in alembic (#77)
.env files don't support shell variable interpolation, so DATABASE_URL
and REDIS_URL had literal ${VAR} strings instead of actual values.
Also alembic/env.py read SQLALCHEMY_DATABASE_URL instead of DATABASE_URL.

Closes #62

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:59:46 -04:00
Patrick Selamy d808819990 fix: resolve alembic migration setup bugs (#62) (#76)
- Replace shell variable interpolation in .env.example with literal
  default values (.env files don't expand variables)
- Change alembic/env.py to read DATABASE_URL instead of
  SQLALCHEMY_DATABASE_URL to match the app's convention

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:58:34 -04:00
Patrick Selamy 20b8ad992c fix: resolve alembic migration setup bugs (#62) (#75)
- Replace shell variable interpolation in .env.example with literal
  default values since .env files don't expand variables
- Change alembic/env.py to read DATABASE_URL instead of
  SQLALCHEMY_DATABASE_URL to match the app's env var name

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:57:33 -04:00
Patrick Selamy bd320f1a8e fix: resolve alembic migration setup bugs (#62) (#74)
- Replace shell variable interpolation in .env.example with literal
  values (.env files don't expand variables)
- Change alembic/env.py to read DATABASE_URL instead of
  SQLALCHEMY_DATABASE_URL to match app configuration

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:56:30 -04:00
Patrick Selamy c65ed2225f fix: resolve alembic migration setup bugs (#62) (#73)
- Replace shell variable interpolation in .env.example with literal
  values (.env files don't expand variables)
- Change alembic/env.py to read DATABASE_URL instead of
  SQLALCHEMY_DATABASE_URL to match app convention

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:55:24 -04:00
Patrick Selamy 06ff48117c fix: use literal values in .env.example and correct env var name in alembic (#72)
.env files don't support shell variable interpolation, so DATABASE_URL
and REDIS_URL had literal ${VAR} strings instead of actual values.
Also alembic/env.py was reading SQLALCHEMY_DATABASE_URL instead of
DATABASE_URL, which is what the app and .env use.

Closes #62

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:54:15 -04:00
Patrick Selamy 34393e7ee3 fix: resolve alembic migration failures from env config bugs (#62) (#71)
- Replace shell variable interpolation in .env.example with literal
  values (.env files don't expand ${VAR} syntax)
- Change alembic/env.py to read DATABASE_URL instead of
  SQLALCHEMY_DATABASE_URL to match the app's env var name

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:53:02 -04:00
Patrick Selamy cf66aac9e3 fix: resolve alembic migration failures from env config bugs (#62) (#70)
- Replace shell variable interpolation in .env.example with literal
  values (.env files don't expand ${VAR} syntax)
- Change alembic/env.py to read DATABASE_URL instead of
  SQLALCHEMY_DATABASE_URL to match the app's env var name

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:52:16 -04:00
Patrick Selamy 13d5945b01 fix: resolve alembic migration failures from env config bugs (#62) (#69)
- Replace shell variable interpolation in .env.example with literal values
- Fix alembic/env.py to read DATABASE_URL instead of SQLALCHEMY_DATABASE_URL

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:51:13 -04:00
Patrick Selamy 7352143004 fix: resolve two bugs preventing alembic migrations (#62) (#68)
1. Replace shell variable interpolation in .env.example with literal
   values, since .env files don't expand variables.
2. Change alembic/env.py to read DATABASE_URL instead of
   SQLALCHEMY_DATABASE_URL to match app convention.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:49:57 -04:00
Patrick Selamy e05ac2ca64 fix: resolve two bugs preventing alembic migrations (#62) (#67)
1. Replace shell variable interpolation in .env.example with literal
   values, since .env files don't expand variables.
2. Change alembic/env.py to read DATABASE_URL instead of
   SQLALCHEMY_DATABASE_URL to match app convention.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:49:18 -04:00
Patrick Selamy 5d533c8c79 fix: resolve two bugs preventing alembic migrations (#62) (#66)
1. Replace shell variable interpolation in .env.example with literal
   values, since .env files don't expand variables.
2. Change alembic/env.py to read DATABASE_URL instead of
   SQLALCHEMY_DATABASE_URL to match app convention.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:48:25 -04:00
Patrick Selamy e67f2d6d8f fix: resolve alembic migration setup bugs (#62) (#65)
- Replace shell variable interpolation in .env.example with literal values
- Change alembic/env.py to read DATABASE_URL instead of SQLALCHEMY_DATABASE_URL

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:47:23 -04:00
Patrick Selamy 387a10132b fix: resolve alembic migration setup bugs (#62) (#64)
- Replace shell variable interpolation in .env.example with literal
  default values (.env files don't expand variables)
- Change alembic/env.py to read DATABASE_URL instead of
  SQLALCHEMY_DATABASE_URL to match the app's env var name

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:46:25 -04:00
Patrick Selamy b044bf25be fix: resolve broken alembic migrations (#62) (#63)
- Replace shell variable interpolation in .env.example with literal
  values, since .env files don't expand shell variables
- Fix alembic/env.py to read DATABASE_URL instead of SQLALCHEMY_DATABASE_URL
  to match the app's environment variable convention

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:45:21 -04:00
16 changed files with 1318 additions and 334 deletions
+2 -2
View File
@@ -6,12 +6,12 @@ POSTGRES_USER=tracker
POSTGRES_PASSWORD=dev_password
# Constructed database URL (for application use)
DATABASE_URL=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@${POSTGRES_HOST}:${POSTGRES_PORT}/${POSTGRES_DB}
DATABASE_URL=postgresql://tracker:dev_password@localhost:5432/polymarket_tracker
# Redis Configuration
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_URL=redis://${REDIS_HOST}:${REDIS_PORT}
REDIS_URL=redis://localhost:6379
# Optional: Development tool ports
ADMINER_PORT=8080
+118 -254
View File
@@ -2,80 +2,103 @@
**Detect informed money before the market moves.**
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![CI](https://github.com/pselamy/polymarket-insider-tracker/actions/workflows/ci.yml/badge.svg)](https://github.com/pselamy/polymarket-insider-tracker/actions/workflows/ci.yml)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
Real-time detection of suspicious trading patterns on Polymarket: fresh wallets, unusual sizing, niche-market activity, and funding chain analysis. Streams trades via WebSocket, profiles wallets on-chain (Polygon), scores risk with ML + heuristics, and dispatches alerts to Discord/Telegram.
---
## The Opportunity
## Quick Start (< 2 minutes)
On January 3, 2026, a trader spotted a significant political event on Polymarket **before it happened**. How? Not by predicting the future, but by tracking suspicious trading behavior.
### 1. Install
> "You don't need to predict the future, you need to track suspicious behavior."
> — [@DidiTrading](https://x.com/DidiTrading)
An insider wallet turned **$35,000 into $442,000** (12.6x return) by entering a position hours before a major market move. The tool that detected this activity flagged five separate alerts before the event occurred.
**This repository builds that tool.**
---
## What This Does
The Polymarket Insider Tracker monitors prediction market trading activity in real-time and identifies patterns that suggest informed trading:
| Signal | What It Detects | Why It Matters |
|--------|-----------------|----------------|
| **Fresh Wallets** | Brand new wallets making large trades | Insiders create new wallets to hide their identity |
| **Unusual Sizing** | Trades that are disproportionately large for the market | Informed traders bet bigger when they have edge |
| **Niche Markets** | Activity in low-volume, specific-outcome markets | Easier to have inside information on obscure events |
| **Funding Chains** | Where wallet funds originated from | Links seemingly separate wallets to the same entity |
When suspicious activity is detected, you receive an instant alert with actionable intelligence.
---
## How It Works
```
┌─────────────────┐ ┌──────────────────┐ ┌────────────────────┐
│ Polymarket API │────>│ Wallet Profiler │────>│ Anomaly Detector │
│ (Real-time) │ │ (Blockchain) │ │ (ML + Heuristics) │
└─────────────────┘ └──────────────────┘ └────────────────────┘
┌────────────────────────────┘
v
┌─────────────────────┐
│ Alert Dispatcher │───> Discord / Telegram / Email
│ "Fresh wallet │
│ buying YES @7.5¢ │
│ on niche market" │
└─────────────────────┘
```bash
# Requires: Python 3.11+, Docker
git clone https://github.com/pselamy/polymarket-insider-tracker.git
cd polymarket-insider-tracker
uv sync --all-extras # or: pip install -e ".[dev]"
```
### Detection Algorithms
### 2. Start infrastructure
1. **Fresh Wallet Detection**
- Checks wallet transaction history on Polygon
- Flags wallets with fewer than 5 lifetime transactions making trades over $1,000
- Traces funding source to identify if connected to known entities
```bash
docker compose up -d # PostgreSQL 15 + Redis 7
docker compose ps # wait for healthy
```
2. **Liquidity Impact Analysis**
- Calculates trade size relative to market depth
- Flags trades consuming more than 2% of visible order book
- Weights by market category (niche markets score higher)
### 3. Configure
3. **Sniper Cluster Detection**
- Uses DBSCAN clustering to find wallets that consistently enter markets within minutes of creation
- Identifies coordinated behavior patterns
```bash
cp .env.example .env
# Edit .env — only DATABASE_URL and REDIS_URL are required for local dev
# (defaults in .env.example work with docker compose)
```
4. **Event Correlation**
- Cross-references trading activity with news feeds
- Detects positions opened 1-4 hours before related news breaks
### 4. Run migrations + start
```bash
uv run alembic upgrade head
uv run python -m polymarket_insider_tracker
```
You should see live trades within seconds:
```
INFO Connection state: disconnected -> connecting
INFO Connected to wss://ws-live-data.polymarket.com and subscribed to trades
DEBUG Trade: BUY 450 @ 1.00 on fifwc-ger-kor-2026-06-14-ger
DEBUG Trade: SELL 5 @ 0.86 on chi1-cd1-cdl-2026-06-14-draw
```
### CLI Options
```bash
python -m polymarket_insider_tracker --help
--version Show version
--config-check Validate configuration and exit
--log-level DEBUG Override log level
--dry-run Run pipeline without sending alerts
--health-port 8080 Override health check port
```
---
## Sample Alert
## Environment Variables
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `DATABASE_URL` | Yes | — | PostgreSQL connection string |
| `REDIS_URL` | No | `redis://localhost:6379` | Redis connection string |
| `POLYGON_RPC_URL` | No | `https://polygon-rpc.com` | Polygon RPC (public default works) |
| `POLYGON_FALLBACK_RPC_URL` | No | — | Fallback RPC endpoint |
| `POLYMARKET_WS_URL` | No | `wss://ws-live-data.polymarket.com` | WebSocket endpoint |
| `POLYMARKET_API_KEY` | No | — | Optional API key for higher rate limits |
| `DISCORD_WEBHOOK_URL` | No | — | Discord alerts |
| `TELEGRAM_BOT_TOKEN` | No | — | Telegram alerts (needs `TELEGRAM_CHAT_ID` too) |
| `TELEGRAM_CHAT_ID` | No | — | Telegram chat for alerts |
| `LOG_LEVEL` | No | `INFO` | Logging level |
| `DRY_RUN` | No | `false` | Skip sending alerts |
| `HEALTH_PORT` | No | `8080` | Health check HTTP port |
No API keys are needed for basic operation — the Polymarket WebSocket and CLOB REST APIs are public.
---
## What It Detects
| Signal | Detection Method | Threshold |
|--------|-----------------|-----------|
| **Fresh Wallets** | Wallet age < 48h, nonce <= 5, making trades > $1k | Confidence 0.5-0.9 |
| **Size Anomalies** | Trade size > 2% of 24h volume or > 5% of order book | Weighted by niche factor |
| **Niche Markets** | Low-volume markets (< $50k daily) with specific outcomes | 1.5x risk multiplier |
| **Funding Chains** | Trace wallet funding to known entities (exchanges, etc.) | On-chain lineage |
| **Sniper Clusters** | DBSCAN clustering of wallets entering within minutes | Coordinated behavior |
Risk scoring combines signals with configurable weights (default threshold: 0.6). Multi-signal bonuses: 2 signals +20%, 3+ signals +30%.
### Sample Alert
```
SUSPICIOUS ACTIVITY DETECTED
@@ -99,207 +122,62 @@ Confidence: HIGH (3/4 signals triggered)
---
## Quick Start
## Architecture
### Prerequisites
```
Polymarket WebSocket ──> Ingestor ──> Profiler ──> Detector ──> Alerter
(wss://ws-live-data) (trades) (on-chain) (scoring) (Discord/TG)
|
Polygon RPC
```
- Python 3.11+
- Docker and Docker Compose
- Polygon RPC endpoint (Alchemy, QuickNode, or self-hosted)
- Polymarket API key (free at [docs.polymarket.com](https://docs.polymarket.com))
### Components
### Installation
| Module | Purpose |
|--------|---------|
| `ingestor/` | WebSocket trade stream + CLOB REST client with rate limiting |
| `profiler/` | Polygon wallet analysis, entity identification, funding chain tracing |
| `detector/` | Fresh wallet, size anomaly, sniper cluster detection, composite risk scorer |
| `alerter/` | Multi-channel dispatch (Discord webhooks, Telegram bot) with dedup |
| `storage/` | SQLAlchemy ORM + Alembic migrations (PostgreSQL) |
| `pipeline.py` | Orchestrator wiring all components together |
| `shutdown.py` | Graceful SIGTERM/SIGINT handling with cleanup callbacks |
---
## Development
```bash
# Clone the repository
git clone https://github.com/pselamy/polymarket-insider-tracker.git
cd polymarket-insider-tracker
# Copy environment template
cp .env.example .env
# Edit .env with your API keys
# Start infrastructure (PostgreSQL, Redis)
docker compose up -d
# Wait for services to be healthy
docker compose ps
# Install Python dependencies
pip install -e .
# Run database migrations
alembic upgrade head
# Run the tracker
python -m src.main
uv run pytest # run tests
uv run ruff check src/ tests/ # lint
uv run ruff format src/ tests/ # format
uv run mypy src/ # type check (strict mode)
```
### Docker Services
The development stack includes:
| Service | Port | Description |
|---------|------|-------------|
| PostgreSQL 15 | 5432 | Primary database |
| Redis 7 | 6379 | Caching and pub/sub |
| Adminer | 8080 | Database admin UI (optional) |
| RedisInsight | 5540 | Redis admin UI (optional) |
```bash
# Start core services only
docker compose up -d
# Start with development tools (Adminer, RedisInsight)
docker compose --profile tools up -d
# View logs
docker compose logs -f
# Stop all services
docker compose down
# Stop and remove volumes (reset data)
docker compose down -v
```
### Configuration
```bash
# .env file
POLYGON_RPC_URL=https://polygon-mainnet.g.alchemy.com/v2/YOUR_KEY
POLYMARKET_API_KEY=your_polymarket_api_key
# Alert destinations (optional)
DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...
TELEGRAM_BOT_TOKEN=your_bot_token
TELEGRAM_CHAT_ID=your_chat_id
# Detection thresholds
MIN_TRADE_SIZE_USDC=1000
FRESH_WALLET_MAX_NONCE=5
LIQUIDITY_IMPACT_THRESHOLD=0.02
```
| Adminer | 8080 | Database admin UI (optional, `--profile tools`) |
| RedisInsight | 5540 | Redis admin UI (optional, `--profile tools`) |
---
## Project Structure
## Troubleshooting
```
polymarket-insider-tracker/
├── src/
│ ├── ingestor/ # Real-time market data ingestion
│ │ ├── clob_client.py # Polymarket CLOB API wrapper
│ │ └── websocket.py # WebSocket event handler
│ ├── profiler/ # Wallet analysis
│ │ ├── analyzer.py # Core wallet profiling logic
│ │ ├── chain.py # Polygon blockchain client
│ │ └── funding.py # Funding chain tracer
│ ├── detector/ # Anomaly detection engines
│ │ ├── fresh_wallet.py
│ │ ├── size_anomaly.py
│ │ ├── sniper.py # DBSCAN clustering
│ │ └── scorer.py # Composite risk scoring
│ ├── alerter/ # Notification dispatch
│ │ ├── formatter.py # Alert message formatting
│ │ └── dispatcher.py # Multi-channel delivery
│ └── storage/ # Persistence layer
│ ├── models.py # SQLAlchemy models
│ └── repos.py # Repository pattern
├── tests/ # Test suite
├── scripts/
│ └── backtest.py # Historical analysis
├── docker-compose.yml
├── pyproject.toml
└── README.md
```
**No trades received / silent connection**
The WebSocket subscription requires `action: "subscribe"` in the envelope. If you're on an older version, update — this was fixed in the WebSocket protocol alignment (see #89).
---
**Connection timeout / DNS errors**
Verify `wss://ws-live-data.polymarket.com` is reachable from your network. Some corporate firewalls block WebSocket connections.
## Roadmap
**Database migration errors**
Ensure PostgreSQL is running (`docker compose ps`) and `DATABASE_URL` matches your docker-compose config. Run `uv run alembic upgrade head` after any schema changes.
### Phase 1: Core Detection (Current)
- [x] Project structure and documentation
- [ ] Polymarket CLOB API integration
- [ ] Fresh wallet detection
- [ ] Size anomaly detection
- [ ] Basic alerting (Discord/Telegram)
### Phase 2: Advanced Intelligence
- [ ] Funding chain analysis
- [ ] Sniper cluster detection (DBSCAN)
- [ ] Market categorization (niche vs mainstream)
- [ ] Historical backtesting framework
### Phase 3: Production Hardening
- [ ] High-availability deployment
- [ ] Rate limit management
- [ ] False positive feedback loop
- [ ] Web dashboard
---
## Why This Matters
Prediction markets are becoming a critical source of real-time probability estimates for world events. As they grow, so does the incentive for informed actors to exploit information asymmetry.
This tool democratizes access to the same detection capabilities that sophisticated traders use. Whether you are:
- **A trader** looking for alpha signals
- **A researcher** studying market microstructure
- **A platform operator** monitoring for manipulation
...this tracker provides visibility into the hidden flows that move markets.
---
## Technical Background
### Polymarket Architecture
Polymarket is a prediction market platform built on Polygon (Ethereum L2). Key characteristics:
- **CLOB (Central Limit Order Book)**: Centralized matching engine for speed
- **On-chain Settlement**: Final trades settle on Polygon blockchain
- **USDC Collateral**: All positions denominated in USDC stablecoin
- **Binary Outcomes**: Shares priced between $0.00 and $1.00
### Data Sources
| Source | Purpose | Latency |
|--------|---------|---------|
| Polymarket CLOB API | Real-time trades, orderbook | Milliseconds |
| Polygon RPC | Wallet history, nonce, funding | 1-2 seconds |
| Market Metadata API | Market categorization | On-demand |
### Detection Challenges
1. **Sybil Resistance**: Insiders use fresh wallets per trade
2. **Rate Limits**: Polygon RPC calls require caching strategy
3. **Market Classification**: NLP needed to categorize market niches
4. **Timing**: CLOB data leads on-chain by seconds
---
## Contributing
Contributions are welcome! Please read our Contributing Guide before submitting PRs.
### Development Setup
```bash
# Install dev dependencies
pip install -e ".[dev]"
# Run tests
pytest
# Run linting
ruff check src/
# Run type checking
mypy src/
```
**Rate limiting on Polygon RPC**
The default public RPC (`https://polygon-rpc.com`) has low rate limits. For production use, set `POLYGON_RPC_URL` to a dedicated provider (Alchemy, QuickNode, etc.).
---
@@ -312,20 +190,6 @@ This software is provided for **educational and research purposes only**.
- Insider trading is illegal in regulated markets; this tool is for transparency and research
- Users are responsible for compliance with applicable laws
---
## License
MIT License - see [LICENSE](LICENSE) for details.
---
## Acknowledgments
- Inspired by [@DidiTrading](https://x.com/DidiTrading) and [@spacexbt](https://x.com/spacexbt)
- Built on the open Polymarket API ecosystem
- Community contributions welcome
---
**Questions?** Open an issue or start a discussion.
+1 -1
View File
@@ -19,7 +19,7 @@ if config.config_file_name is not None:
target_metadata = Base.metadata
# Get database URL from environment variable or config
database_url = os.environ.get("SQLALCHEMY_DATABASE_URL")
database_url = os.environ.get("DATABASE_URL")
if database_url:
config.set_main_option("sqlalchemy.url", database_url)
@@ -5,16 +5,16 @@ Revises:
Create Date: 2026-01-04 00:00:00.000000+00:00
"""
from typing import Sequence, Union
from collections.abc import Sequence
import sqlalchemy as sa
from alembic import op
# revision identifiers, used by Alembic.
revision: str = "001_initial"
down_revision: Union[str, None] = None
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None
down_revision: str | None = None
branch_labels: str | Sequence[str] | None = None
depends_on: str | Sequence[str] | None = None
def upgrade() -> None:
-2
View File
@@ -1,5 +1,3 @@
version: "3.8"
services:
postgres:
image: postgres:15
@@ -0,0 +1,113 @@
# Skill: tracking-prediction-market-flow
Use when analyzing prediction market activity for informed-flow signals, insider
trading patterns, or suspicious wallet behavior on Polymarket.
## What This Tool Does
polymarket-insider-tracker streams real-time trades from Polymarket's WebSocket
feed, profiles trader wallets on the Polygon blockchain, and scores each trade
for informed-flow risk using multiple detection signals:
- **Fresh wallet detection**: New wallets (age < 48h, nonce <= 5) making large
trades (> $1k). Insiders create disposable wallets per trade.
- **Size anomaly detection**: Trades consuming > 2% of 24h volume or > 5% of
visible order book depth. Informed traders bet bigger when they have edge.
- **Niche market scoring**: Low-volume markets (< $50k daily) get a 1.5x risk
multiplier. Easier to have inside information on obscure events.
- **Funding chain analysis**: Traces wallet funding sources on-chain to link
seemingly separate wallets to the same entity or exchange.
- **Sniper cluster detection**: DBSCAN clustering identifies wallets that
consistently enter markets within minutes of creation.
Composite risk scoring combines signals with configurable weights (default
alert threshold: 0.6). Multi-signal bonuses: 2 signals +20%, 3+ signals +30%.
## Installation
```bash
git clone https://github.com/pselamy/polymarket-insider-tracker.git
cd polymarket-insider-tracker
uv sync --all-extras
docker compose up -d # PostgreSQL + Redis
cp .env.example .env # defaults work for local dev
uv run alembic upgrade head
```
No API keys required for basic operation (Polymarket APIs are public).
## Usage
```bash
# Start the tracker (streams trades, profiles wallets, scores risk, alerts)
uv run python -m polymarket_insider_tracker
# Dry run (no alerts sent)
uv run python -m polymarket_insider_tracker --dry-run
# Debug mode (see every trade)
uv run python -m polymarket_insider_tracker --log-level DEBUG
# Validate config without starting
uv run python -m polymarket_insider_tracker --config-check
```
## Interpreting Signals
### Risk Assessment Output
Each flagged trade produces a risk assessment with:
- **Confidence score** (0.0-1.0): Composite of weighted signals
- **Signal breakdown**: Which detectors fired and their individual confidence
- **Wallet profile**: Age, nonce, transaction count, funding source
- **Market context**: Volume, category, order book depth
### Signal Interpretation Guide
| Score Range | Interpretation | Action |
|-------------|---------------|--------|
| 0.6-0.7 | Moderate: single strong signal or two weak ones | Monitor, note the market |
| 0.7-0.85 | High: multiple signals converging | Investigate the market and wallet |
| 0.85-1.0 | Critical: fresh wallet + large size + niche market | High-confidence informed flow |
### What This Is NOT
- Not a trading signal generator. Informed flow != actionable alpha without
further analysis (hypothesis -> leakage-aware backtest -> capital).
- Not real-time enough for front-running. The tool detects patterns for
research and monitoring, not millisecond-level execution.
- Detection of informed flow does not prove insider trading. Many legitimate
reasons exist for the patterns this tool flags.
## Rate Limits and Etiquette
- **Polymarket WebSocket**: No explicit rate limit; one persistent connection.
Do not open multiple connections unnecessarily.
- **Polymarket CLOB REST**: Built-in rate limiter at 10 req/s with retry
backoff on 429/5xx. Respect this for metadata/orderbook queries.
- **Polygon RPC**: Public endpoints (polygon-rpc.com) have low limits. For
sustained use, configure a dedicated RPC provider via `POLYGON_RPC_URL`.
Built-in token-bucket rate limiter at 25 req/s with Redis caching (5min TTL).
## Known Pitfalls
1. **WebSocket subscription format**: Must include `action: "subscribe"` in the
envelope. Without it, the server accepts the connection but delivers zero
trade events (silent failure). Fixed in the current version.
2. **Message routing**: Live-data WebSocket pushes `{connection_id, payload:
{...trade fields}}`, not `{topic, type, payload}`. Route by checking for
`transactionHash` + `proxyWallet` keys in `payload`.
3. **Public RPC rate limits**: Default Polygon RPC will throttle under load.
Use a dedicated provider for production.
4. **Database required**: PostgreSQL + Redis must be running. Use
`docker compose up -d` for local dev.
## Cross-References
- **Repository**: https://github.com/pselamy/polymarket-insider-tracker
- **Issues**: https://github.com/pselamy/polymarket-insider-tracker/issues
- **Agent skill landing** (follow-on): selamy-labs/agent-skills
+25 -8
View File
@@ -18,7 +18,9 @@ from pydantic_settings import BaseSettings, SettingsConfigDict
class DatabaseSettings(BaseSettings):
"""Database connection settings."""
model_config = SettingsConfigDict(env_prefix="")
model_config = SettingsConfigDict(
env_prefix="", env_file=".env", env_file_encoding="utf-8", extra="ignore"
)
url: str = Field(
alias="DATABASE_URL",
@@ -37,7 +39,9 @@ class DatabaseSettings(BaseSettings):
class RedisSettings(BaseSettings):
"""Redis connection settings."""
model_config = SettingsConfigDict(env_prefix="")
model_config = SettingsConfigDict(
env_prefix="", env_file=".env", env_file_encoding="utf-8", extra="ignore"
)
url: str = Field(
default="redis://localhost:6379",
@@ -57,7 +61,9 @@ class RedisSettings(BaseSettings):
class PolygonSettings(BaseSettings):
"""Polygon blockchain RPC settings."""
model_config = SettingsConfigDict(env_prefix="POLYGON_")
model_config = SettingsConfigDict(
env_prefix="POLYGON_", env_file=".env", env_file_encoding="utf-8", extra="ignore"
)
rpc_url: str = Field(
default="https://polygon-rpc.com",
@@ -84,7 +90,9 @@ class PolygonSettings(BaseSettings):
class PolymarketSettings(BaseSettings):
"""Polymarket API settings."""
model_config = SettingsConfigDict(env_prefix="POLYMARKET_")
model_config = SettingsConfigDict(
env_prefix="POLYMARKET_", env_file=".env", env_file_encoding="utf-8", extra="ignore"
)
ws_url: str = Field(
default="wss://ws-subscriptions-clob.polymarket.com/ws/market",
@@ -109,7 +117,9 @@ class PolymarketSettings(BaseSettings):
class DiscordSettings(BaseSettings):
"""Discord notification settings."""
model_config = SettingsConfigDict(env_prefix="DISCORD_")
model_config = SettingsConfigDict(
env_prefix="DISCORD_", env_file=".env", env_file_encoding="utf-8", extra="ignore"
)
webhook_url: SecretStr | None = Field(
default=None,
@@ -120,13 +130,15 @@ class DiscordSettings(BaseSettings):
@property
def enabled(self) -> bool:
"""Check if Discord notifications are enabled."""
return self.webhook_url is not None
return self.webhook_url is not None and bool(self.webhook_url.get_secret_value().strip())
class TelegramSettings(BaseSettings):
"""Telegram notification settings."""
model_config = SettingsConfigDict(env_prefix="TELEGRAM_")
model_config = SettingsConfigDict(
env_prefix="TELEGRAM_", env_file=".env", env_file_encoding="utf-8", extra="ignore"
)
bot_token: SecretStr | None = Field(
default=None,
@@ -142,7 +154,12 @@ class TelegramSettings(BaseSettings):
@property
def enabled(self) -> bool:
"""Check if Telegram notifications are enabled."""
return self.bot_token is not None and self.chat_id is not None
return (
self.bot_token is not None
and bool(self.bot_token.get_secret_value().strip())
and self.chat_id is not None
and bool(self.chat_id.strip())
)
class Settings(BaseSettings):
@@ -0,0 +1,205 @@
"""Gamma API client for Polymarket market volume / liquidity data.
The CLOB API does not expose 24h volume or liquidity. The public Gamma API
(https://gamma-api.polymarket.com) does, with no auth required. This module
fetches the volume/liquidity snapshot keyed by condition_id so the
size_anomaly detector can do real ratio math instead of falling back to
the niche-base 0.2 confidence floor.
"""
from __future__ import annotations
import asyncio
import logging
from dataclasses import dataclass
from decimal import Decimal, InvalidOperation
import httpx
logger = logging.getLogger(__name__)
DEFAULT_HOST = "https://gamma-api.polymarket.com"
DEFAULT_TIMEOUT_SECONDS = 15.0
# Gamma /markets enforces a server-side max of 100 per page even when a
# higher `limit` is sent. Using 100 lines our page size up with the actual
# response so pagination doesn't bail out after the first page.
DEFAULT_PAGE_LIMIT = 100
# Gamma also caps `offset` around 10000 for this collection. Combined with
# the 100/page limit that gives ~10k markets max, sequential — way too slow
# at default sync interval. We sort by 24h volume desc and only walk the
# top N pages, since markets with zero recent volume don't need a real
# ratio anyway (the niche path handles them).
DEFAULT_MAX_PAGES = 50 # 50 * 100 = 5000 most-traded markets per sync
DEFAULT_PAGE_CONCURRENCY = 5
DEFAULT_MAX_RETRIES = 3
DEFAULT_RETRY_BASE_DELAY_SECONDS = 1.0
@dataclass(frozen=True)
class GammaMarketStats:
"""Volume / liquidity snapshot for a single market from gamma-api."""
condition_id: str
daily_volume: Decimal | None
weekly_volume: Decimal | None
monthly_volume: Decimal | None
total_volume: Decimal | None
liquidity: Decimal | None
def _to_decimal(value: object) -> Decimal | None:
if value is None or value == "":
return None
try:
return Decimal(str(value))
except (InvalidOperation, ValueError, TypeError):
return None
def _parse_market(raw: dict[str, object]) -> GammaMarketStats | None:
cid = raw.get("conditionId")
if not cid or not isinstance(cid, str):
return None
return GammaMarketStats(
condition_id=cid,
daily_volume=_to_decimal(raw.get("volume24hr")),
weekly_volume=_to_decimal(raw.get("volume1wk")),
monthly_volume=_to_decimal(raw.get("volume1mo")),
total_volume=_to_decimal(raw.get("volumeNum") or raw.get("volume")),
liquidity=_to_decimal(raw.get("liquidityNum") or raw.get("liquidity")),
)
class GammaClientError(Exception):
"""Raised when gamma-api returns an unrecoverable error."""
class GammaClient:
"""Async client for the public gamma-api markets endpoint.
Provides batched, paginated reads of every active market with their
24h/weekly/monthly volume and current liquidity. Designed to be called
from MarketMetadataSync once per sync interval; results are merged into
Redis-cached MarketMetadata objects.
"""
def __init__(
self,
*,
host: str = DEFAULT_HOST,
timeout_seconds: float = DEFAULT_TIMEOUT_SECONDS,
page_limit: int = DEFAULT_PAGE_LIMIT,
max_pages: int = DEFAULT_MAX_PAGES,
page_concurrency: int = DEFAULT_PAGE_CONCURRENCY,
max_retries: int = DEFAULT_MAX_RETRIES,
retry_base_delay_seconds: float = DEFAULT_RETRY_BASE_DELAY_SECONDS,
) -> None:
self._host = host.rstrip("/")
self._timeout = timeout_seconds
self._page_limit = page_limit
self._max_pages = max_pages
self._page_concurrency = page_concurrency
self._max_retries = max_retries
self._retry_base = retry_base_delay_seconds
async def _get_with_retry(
self,
client: httpx.AsyncClient,
path: str,
params: dict[str, object],
) -> list[dict[str, object]]:
last_exc: Exception | None = None
delay = self._retry_base
for attempt in range(self._max_retries):
try:
resp = await client.get(path, params=params)
resp.raise_for_status()
payload = resp.json()
if not isinstance(payload, list):
raise GammaClientError(
f"Unexpected gamma response shape for {path}: {type(payload).__name__}"
)
return payload
except (httpx.HTTPError, ValueError) as exc:
last_exc = exc
logger.warning(
"gamma %s attempt %d/%d failed: %s",
path,
attempt + 1,
self._max_retries,
exc,
)
if attempt < self._max_retries - 1:
await asyncio.sleep(delay)
delay *= 2
raise GammaClientError(
f"gamma {path} failed after {self._max_retries} attempts: {last_exc}"
)
async def get_active_market_stats(self) -> dict[str, GammaMarketStats]:
"""Fetch volume/liquidity for the most-traded active markets.
Walks up to `max_pages` pages of `page_limit` markets each, sorted
by 24h volume descending, with bounded concurrency. Markets beyond
that window have effectively zero recent volume — the size_anomaly
niche path handles them without needing a ratio.
Returns:
Mapping condition_id -> GammaMarketStats.
"""
results: dict[str, GammaMarketStats] = {}
sem = asyncio.Semaphore(self._page_concurrency)
stop = asyncio.Event()
async with httpx.AsyncClient(
base_url=self._host,
timeout=self._timeout,
headers={"User-Agent": "polymarket-insider-tracker/0.1"},
) as client:
async def fetch_page(page_index: int) -> list[dict[str, object]]:
if stop.is_set():
return []
params = {
"limit": self._page_limit,
"offset": page_index * self._page_limit,
"active": "true",
"closed": "false",
"order": "volume24hr",
"ascending": "false",
}
async with sem:
if stop.is_set():
return []
try:
return await self._get_with_retry(client, "/markets", params)
except GammaClientError as exc:
# Gamma rejects offsets past its hard cap with a
# validation error; treat that as a clean stop.
logger.debug("gamma stop at page %d: %s", page_index, exc)
stop.set()
return []
tasks = [asyncio.create_task(fetch_page(i)) for i in range(self._max_pages)]
pages = await asyncio.gather(*tasks)
empty_streak = 0
for page in pages:
if not page:
empty_streak += 1
continue
empty_streak = 0
for raw in page:
if not isinstance(raw, dict):
continue
parsed = _parse_market(raw)
if parsed is not None:
results[parsed.condition_id] = parsed
if len(page) < self._page_limit:
# short page — we walked past the end of the active set
empty_streak += 1
if empty_streak >= 2:
break
logger.info("gamma sync: fetched stats for %d active markets", len(results))
return results
@@ -9,13 +9,14 @@ import contextlib
import json
import logging
from collections.abc import Callable
from dataclasses import dataclass
from dataclasses import dataclass, replace
from datetime import UTC, datetime
from enum import Enum
from enum import StrEnum
from redis.asyncio import Redis
from .clob_client import ClobClient
from .gamma_client import GammaClient, GammaClientError, GammaMarketStats
from .models import MarketMetadata
logger = logging.getLogger(__name__)
@@ -27,7 +28,7 @@ DEFAULT_CACHE_TTL_SECONDS = 600 # 10 minutes
DEFAULT_REDIS_KEY_PREFIX = "polymarket:market:"
class SyncState(str, Enum):
class SyncState(StrEnum):
"""State of the metadata synchronizer."""
STOPPED = "stopped"
@@ -91,6 +92,7 @@ class MarketMetadataSync:
redis: Redis,
clob_client: ClobClient,
*,
gamma_client: GammaClient | None = None,
sync_interval_seconds: int = DEFAULT_SYNC_INTERVAL_SECONDS,
cache_ttl_seconds: int = DEFAULT_CACHE_TTL_SECONDS,
key_prefix: str = DEFAULT_REDIS_KEY_PREFIX,
@@ -102,6 +104,8 @@ class MarketMetadataSync:
Args:
redis: Redis async client for caching.
clob_client: CLOB API client for fetching markets.
gamma_client: Optional gamma-api client for volume/liquidity
enrichment. Defaults to a fresh GammaClient() instance.
sync_interval_seconds: Interval between syncs (default: 300 / 5 min).
cache_ttl_seconds: TTL for cached entries (default: 600 / 10 min).
key_prefix: Redis key prefix for market data.
@@ -110,6 +114,7 @@ class MarketMetadataSync:
"""
self._redis = redis
self._clob = clob_client
self._gamma = gamma_client or GammaClient()
self._sync_interval = sync_interval_seconds
self._cache_ttl = cache_ttl_seconds
self._key_prefix = key_prefix
@@ -216,6 +221,18 @@ class MarketMetadataSync:
self._set_state(SyncState.ERROR)
# Continue running - will retry on next interval
async def _fetch_gamma_stats(self) -> dict[str, GammaMarketStats]:
"""Fetch volume/liquidity stats from gamma-api.
Returns an empty dict on failure so a degraded gamma endpoint
does not stop CLOB metadata from being cached.
"""
try:
return await self._gamma.get_active_market_stats()
except (GammaClientError, Exception) as e:
logger.warning("gamma stats fetch failed (continuing without volume): %s", e)
return {}
async def _sync_all_markets(self) -> None:
"""Fetch all markets and cache them in Redis."""
self._set_state(SyncState.SYNCING)
@@ -223,14 +240,27 @@ class MarketMetadataSync:
self._stats.total_syncs += 1
try:
# Fetch markets from CLOB API (runs in thread pool for sync API)
markets = await asyncio.to_thread(self._clob.get_markets, True)
# Fetch CLOB markets and gamma volume snapshot in parallel
markets, gamma_stats = await asyncio.gather(
asyncio.to_thread(self._clob.get_markets, True),
self._fetch_gamma_stats(),
)
# Cache each market in Redis
# Cache each market in Redis, enriched with gamma volume/liquidity
cached_count = 0
enriched_count = 0
for market in markets:
try:
metadata = MarketMetadata.from_market(market)
stats = gamma_stats.get(metadata.condition_id)
if stats is not None:
metadata = replace(
metadata,
daily_volume=stats.daily_volume,
weekly_volume=stats.weekly_volume,
liquidity=stats.liquidity,
)
enriched_count += 1
await self._cache_market(metadata)
cached_count += 1
except Exception as e:
@@ -246,7 +276,10 @@ class MarketMetadataSync:
self._set_state(SyncState.IDLE)
logger.info(
f"Synced {cached_count} markets in {self._stats.last_sync_duration_seconds:.2f}s"
"Synced %d markets (%d enriched with gamma volume) in %.2fs",
cached_count,
enriched_count,
self._stats.last_sync_duration_seconds,
)
# Notify callback
@@ -86,7 +86,7 @@ class Orderbook:
bids: tuple[OrderbookLevel, ...]
asks: tuple[OrderbookLevel, ...]
tick_size: Decimal
timestamp: datetime = field(default_factory=datetime.utcnow)
timestamp: datetime = field(default_factory=lambda: datetime.now(UTC))
@classmethod
def from_clob_orderbook(cls, orderbook: Any) -> "Orderbook":
@@ -399,6 +399,12 @@ class MarketMetadata:
# Derived metadata
category: str = "other"
# Liquidity/volume snapshot (from gamma-api). All optional — older
# cache entries and CLOB-only sync results may not have these.
daily_volume: Decimal | None = None
weekly_volume: Decimal | None = None
liquidity: Decimal | None = None
# Cache metadata
last_updated: datetime = field(default_factory=lambda: datetime.now(UTC))
@@ -446,6 +452,9 @@ class MarketMetadata:
"active": self.active,
"closed": self.closed,
"category": self.category,
"daily_volume": str(self.daily_volume) if self.daily_volume is not None else None,
"weekly_volume": str(self.weekly_volume) if self.weekly_volume is not None else None,
"liquidity": str(self.liquidity) if self.liquidity is not None else None,
"last_updated": self.last_updated.isoformat(),
}
@@ -477,6 +486,15 @@ class MarketMetadata:
else:
last_updated = datetime.now(UTC)
def _opt_dec(key: str) -> Decimal | None:
raw = data.get(key)
if raw is None or raw == "":
return None
try:
return Decimal(str(raw))
except (ValueError, ArithmeticError):
return None
return cls(
condition_id=str(data["condition_id"]),
question=str(data.get("question", "")),
@@ -486,5 +504,8 @@ class MarketMetadata:
active=bool(data.get("active", True)),
closed=bool(data.get("closed", False)),
category=str(data.get("category", "other")),
daily_volume=_opt_dec("daily_volume"),
weekly_volume=_opt_dec("weekly_volume"),
liquidity=_opt_dec("liquidity"),
last_updated=last_updated,
)
@@ -9,8 +9,9 @@ from dataclasses import dataclass
from enum import Enum
from typing import Any
import websockets
from websockets.asyncio.client import ClientConnection
from websockets.asyncio.client import connect as ws_connect
from websockets.exceptions import ConnectionClosed
from polymarket_insider_tracker.ingestor.models import TradeEvent
@@ -149,14 +150,14 @@ class TradeStreamHandler:
elif self._market_filter:
subscription["filters"] = json.dumps({"market_slug": self._market_filter})
return {"subscriptions": [subscription]}
return {"action": "subscribe", "subscriptions": [subscription]}
async def _connect(self) -> ClientConnection:
"""Establish WebSocket connection."""
await self._set_state(ConnectionState.CONNECTING)
try:
ws = await websockets.connect(
ws = await ws_connect(
self._host,
ping_interval=self._ping_interval,
ping_timeout=self._ping_interval * 2,
@@ -182,12 +183,13 @@ class TradeStreamHandler:
try:
data = json.loads(message)
# Check if this is a trade message
topic = data.get("topic")
msg_type = data.get("type")
if topic == "activity" and msg_type == "trades":
payload = data.get("payload", {})
# ws-live-data pushes {connection_id, payload:{...trade fields}}
payload = data.get("payload")
if (
isinstance(payload, dict)
and "transactionHash" in payload
and "proxyWallet" in payload
):
trade = TradeEvent.from_websocket_message(payload)
self._stats.trades_received += 1
@@ -207,8 +209,7 @@ class TradeStreamHandler:
logger.error("Error in trade callback: %s", e)
else:
# Log other message types for debugging
logger.debug("Received message: topic=%s type=%s", topic, msg_type)
logger.debug("Received non-trade message: %s", str(data)[:120])
except json.JSONDecodeError as e:
logger.warning("Invalid JSON message: %s", e)
@@ -227,7 +228,7 @@ class TradeStreamHandler:
else:
logger.debug("Received binary message (%d bytes)", len(message))
except websockets.ConnectionClosed as e:
except ConnectionClosed as e:
logger.warning("Connection closed: %s", e)
raise
except Exception as e:
@@ -284,7 +285,7 @@ class TradeStreamHandler:
while self._running:
try:
await self._listen(self._ws)
except (websockets.ConnectionClosed, Exception) as e:
except (ConnectionClosed, Exception) as e:
if not self._running:
break
+75 -2
View File
@@ -11,7 +11,7 @@ import contextlib
import logging
from dataclasses import dataclass
from datetime import UTC, datetime
from enum import Enum
from enum import StrEnum
from typing import TYPE_CHECKING
from redis.asyncio import Redis
@@ -29,7 +29,14 @@ from polymarket_insider_tracker.ingestor.metadata_sync import MarketMetadataSync
from polymarket_insider_tracker.ingestor.websocket import TradeStreamHandler
from polymarket_insider_tracker.profiler.analyzer import WalletAnalyzer
from polymarket_insider_tracker.profiler.chain import PolygonClient
from polymarket_insider_tracker.profiler.funding import FundingTracer
from polymarket_insider_tracker.storage.database import DatabaseManager
from polymarket_insider_tracker.storage.repos import (
FundingRepository,
FundingTransferDTO,
WalletProfileDTO,
WalletRepository,
)
if TYPE_CHECKING:
from typing import Any
@@ -43,7 +50,7 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
class PipelineState(str, Enum):
class PipelineState(StrEnum):
"""Pipeline lifecycle states."""
STOPPED = "stopped"
@@ -120,6 +127,7 @@ class Pipeline:
self._alert_formatter: AlertFormatter | None = None
self._alert_dispatcher: AlertDispatcher | None = None
self._trade_stream: TradeStreamHandler | None = None
self._funding_tracer: FundingTracer | None = None
# Synchronization
self._stop_event: asyncio.Event | None = None
@@ -233,6 +241,10 @@ class Pipeline:
redis=self._redis,
)
# Initialize Funding Tracer
logger.debug("Initializing funding tracer...")
self._funding_tracer = FundingTracer(self._polygon_client)
# Initialize Detectors
logger.debug("Initializing detectors...")
self._fresh_wallet_detector = FreshWalletDetector(self._wallet_analyzer)
@@ -365,6 +377,10 @@ class Pipeline:
self._detect_size_anomaly(trade),
)
# Persist wallet profile and funding data when a fresh wallet is detected
if fresh_signal is not None:
await self._persist_wallet_and_funding(fresh_signal)
# Bundle signals
bundle = SignalBundle(
trade_event=trade,
@@ -382,6 +398,63 @@ class Pipeline:
self._stats.errors += 1
self._stats.last_error = str(e)
async def _persist_wallet_and_funding(self, signal: FreshWalletSignal) -> None:
"""Persist wallet profile and funding transfers to Postgres.
Called when a fresh wallet signal is detected. Upserts the wallet
profile and traces/inserts any funding transfers found on-chain.
Args:
signal: The fresh wallet signal containing the wallet profile.
"""
if not self._db_manager:
return
profile = signal.wallet_profile
address = profile.address
try:
async with self._db_manager.get_async_session() as session:
# Persist wallet profile
wallet_repo = WalletRepository(session)
dto = WalletProfileDTO(
address=address,
nonce=profile.nonce,
first_seen_at=profile.first_seen,
is_fresh=profile.is_fresh,
matic_balance=profile.matic_balance,
usdc_balance=profile.usdc_balance,
analyzed_at=profile.analyzed_at,
)
await wallet_repo.upsert(dto)
# Trace and persist funding transfers
if self._funding_tracer:
chain = await self._funding_tracer.trace(address)
if chain.chain:
funding_repo = FundingRepository(session)
funding_dtos = [
FundingTransferDTO(
from_address=t.from_address,
to_address=t.to_address,
amount=t.amount,
token=t.token,
tx_hash=t.tx_hash,
block_number=t.block_number,
timestamp=t.timestamp,
)
for t in chain.chain
]
await funding_repo.insert_many(funding_dtos)
logger.debug(
"Persisted wallet profile and %d funding transfers for %s",
len(chain.chain) if self._funding_tracer and chain.chain else 0,
address[:10] + "...",
)
except Exception as e:
logger.warning("Failed to persist wallet/funding data for %s: %s", address, e)
async def _detect_fresh_wallet(self, trade: TradeEvent) -> FreshWalletSignal | None:
"""Run fresh wallet detection."""
if not self._fresh_wallet_detector:
+249
View File
@@ -0,0 +1,249 @@
"""Tests for the gamma-api client."""
from __future__ import annotations
from decimal import Decimal
import httpx
import pytest
from polymarket_insider_tracker.ingestor import gamma_client as gamma_module
from polymarket_insider_tracker.ingestor.gamma_client import (
GammaClient,
GammaClientError,
GammaMarketStats,
_parse_market,
)
class TestParseMarket:
def test_parses_full_payload(self) -> None:
raw = {
"conditionId": "0xabc",
"volume24hr": "12345.67",
"volume1wk": "100000",
"volume1mo": "500000",
"volumeNum": "999999.5",
"liquidityNum": "42000",
}
stats = _parse_market(raw)
assert stats is not None
assert stats.condition_id == "0xabc"
assert stats.daily_volume == Decimal("12345.67")
assert stats.weekly_volume == Decimal("100000")
assert stats.monthly_volume == Decimal("500000")
assert stats.total_volume == Decimal("999999.5")
assert stats.liquidity == Decimal("42000")
def test_falls_back_to_alternative_keys(self) -> None:
raw = {
"conditionId": "0x1",
"volume24hr": "1",
"volume": "777",
"liquidity": "55",
}
stats = _parse_market(raw)
assert stats is not None
assert stats.total_volume == Decimal("777")
assert stats.liquidity == Decimal("55")
def test_handles_missing_numeric_fields(self) -> None:
stats = _parse_market({"conditionId": "0x2"})
assert stats is not None
assert stats.daily_volume is None
assert stats.liquidity is None
def test_drops_garbage_decimals(self) -> None:
stats = _parse_market(
{"conditionId": "0x3", "volume24hr": "not-a-number", "liquidityNum": ""}
)
assert stats is not None
assert stats.daily_volume is None
assert stats.liquidity is None
def test_rejects_missing_condition_id(self) -> None:
assert _parse_market({"volume24hr": "1"}) is None
assert _parse_market({"conditionId": ""}) is None
assert _parse_market({"conditionId": 123}) is None # type: ignore[arg-type]
def _make_client(
_handler: httpx.MockTransport,
*,
page_limit: int = 100,
max_pages: int = 5,
page_concurrency: int = 5,
max_retries: int = 1,
) -> GammaClient:
"""Build a GammaClient that constructs httpx.AsyncClient with the given transport.
GammaClient creates its own AsyncClient inside `get_active_market_stats`,
so we monkeypatch the AsyncClient factory in the module to inject the mock
transport.
"""
return GammaClient(
page_limit=page_limit,
max_pages=max_pages,
page_concurrency=page_concurrency,
max_retries=max_retries,
retry_base_delay_seconds=0.0,
)
@pytest.fixture
def patch_async_client(monkeypatch: pytest.MonkeyPatch):
"""Replace the AsyncClient used by gamma_client with one bound to a MockTransport."""
def _apply(handler: httpx.MockTransport) -> None:
original = gamma_module.httpx.AsyncClient
def factory(*args: object, **kwargs: object) -> httpx.AsyncClient:
kwargs["transport"] = handler # type: ignore[index]
return original(*args, **kwargs) # type: ignore[arg-type]
monkeypatch.setattr(gamma_module.httpx, "AsyncClient", factory)
return _apply
@pytest.mark.asyncio
async def test_get_active_market_stats_single_page(patch_async_client) -> None:
page_one = [
{"conditionId": "0xa", "volume24hr": "100", "liquidityNum": "10"},
{"conditionId": "0xb", "volume24hr": "200", "liquidityNum": "20"},
]
calls: list[dict[str, str]] = []
def handler(request: httpx.Request) -> httpx.Response:
calls.append(dict(request.url.params))
offset = int(request.url.params.get("offset", "0"))
if offset == 0:
return httpx.Response(200, json=page_one)
return httpx.Response(200, json=[])
transport = httpx.MockTransport(handler)
patch_async_client(transport)
client = _make_client(transport, page_limit=2, max_pages=3)
result = await client.get_active_market_stats()
assert set(result.keys()) == {"0xa", "0xb"}
assert isinstance(result["0xa"], GammaMarketStats)
assert result["0xa"].daily_volume == Decimal("100")
assert calls[0]["limit"] == "2"
assert calls[0]["order"] == "volume24hr"
assert calls[0]["ascending"] == "false"
@pytest.mark.asyncio
async def test_get_active_market_stats_short_page_stops(patch_async_client) -> None:
"""A page shorter than page_limit signals end-of-data after a small empty streak."""
page_zero = [{"conditionId": f"0x{i}", "volume24hr": str(i)} for i in range(5)]
page_one_short = [{"conditionId": "0xshort", "volume24hr": "1"}]
def handler(request: httpx.Request) -> httpx.Response:
offset = int(request.url.params.get("offset", "0"))
if offset == 0:
return httpx.Response(200, json=page_zero)
if offset == 5:
return httpx.Response(200, json=page_one_short)
return httpx.Response(200, json=[])
transport = httpx.MockTransport(handler)
patch_async_client(transport)
client = _make_client(transport, page_limit=5, max_pages=10)
result = await client.get_active_market_stats()
assert "0xshort" in result
assert len(result) == 6
@pytest.mark.asyncio
async def test_get_active_market_stats_offset_cap_clean_stop(
patch_async_client,
) -> None:
"""Gamma rejects offsets past its hard cap; that error is swallowed cleanly."""
def handler(request: httpx.Request) -> httpx.Response:
offset = int(request.url.params.get("offset", "0"))
if offset == 0:
return httpx.Response(200, json=[{"conditionId": "0xa", "volume24hr": "1"}])
return httpx.Response(400, json={"error": "offset too large"})
transport = httpx.MockTransport(handler)
patch_async_client(transport)
client = _make_client(transport, page_limit=1, max_pages=4, max_retries=1)
result = await client.get_active_market_stats()
assert "0xa" in result
@pytest.mark.asyncio
async def test_get_with_retry_recovers_after_transient_error(
patch_async_client,
) -> None:
"""Transient HTTP errors retry up to max_retries before giving up."""
state = {"attempts": 0}
def handler(request: httpx.Request) -> httpx.Response:
offset = int(request.url.params.get("offset", "0"))
if offset == 0:
state["attempts"] += 1
if state["attempts"] < 2:
return httpx.Response(503, json={"error": "transient"})
return httpx.Response(200, json=[{"conditionId": "0xrecover", "volume24hr": "1"}])
return httpx.Response(200, json=[])
transport = httpx.MockTransport(handler)
patch_async_client(transport)
client = _make_client(transport, page_limit=1, max_pages=2, max_retries=3)
result = await client.get_active_market_stats()
assert "0xrecover" in result
assert state["attempts"] == 2
@pytest.mark.asyncio
async def test_unexpected_response_shape_is_handled(patch_async_client) -> None:
"""A non-list payload becomes a clean stop, not a crash."""
def handler(_request: httpx.Request) -> httpx.Response:
return httpx.Response(200, json={"unexpected": "shape"})
transport = httpx.MockTransport(handler)
patch_async_client(transport)
client = _make_client(transport, page_limit=1, max_pages=2, max_retries=1)
result = await client.get_active_market_stats()
assert result == {}
@pytest.mark.asyncio
async def test_skips_non_dict_entries(patch_async_client) -> None:
"""Defensive: server returning mixed-type list items shouldn't crash."""
def handler(request: httpx.Request) -> httpx.Response:
offset = int(request.url.params.get("offset", "0"))
if offset == 0:
return httpx.Response(
200,
json=[
{"conditionId": "0xa", "volume24hr": "1"},
"garbage",
None,
42,
],
)
return httpx.Response(200, json=[])
transport = httpx.MockTransport(handler)
patch_async_client(transport)
client = _make_client(transport, page_limit=4, max_pages=2)
result = await client.get_active_market_stats()
assert list(result.keys()) == ["0xa"]
def test_gamma_client_error_inherits_exception() -> None:
assert issubclass(GammaClientError, Exception)
+67 -26
View File
@@ -8,6 +8,7 @@ from unittest.mock import AsyncMock, MagicMock
import pytest
from polymarket_insider_tracker.ingestor.clob_client import ClobClient
from polymarket_insider_tracker.ingestor.gamma_client import GammaClient
from polymarket_insider_tracker.ingestor.metadata_sync import (
DEFAULT_CACHE_TTL_SECONDS,
DEFAULT_REDIS_KEY_PREFIX,
@@ -72,6 +73,18 @@ def mock_clob(sample_market: Market) -> MagicMock:
return clob
@pytest.fixture
def mock_gamma() -> MagicMock:
"""Create a mock GammaClient that returns empty volume stats.
Without this, MarketMetadataSync would instantiate a default
GammaClient and hit the real gamma-api over HTTP during unit tests.
"""
gamma = MagicMock(spec=GammaClient)
gamma.get_active_market_stats = AsyncMock(return_value={})
return gamma
class TestDeriveCategory:
"""Tests for the derive_category function."""
@@ -195,9 +208,9 @@ class TestSyncStats:
class TestMarketMetadataSync:
"""Tests for the MarketMetadataSync class."""
def test_init(self, mock_redis: AsyncMock, mock_clob: MagicMock) -> None:
def test_init(self, mock_redis: AsyncMock, mock_clob: MagicMock, mock_gamma: MagicMock) -> None:
"""Test initialization."""
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob)
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob, gamma_client=mock_gamma)
assert sync.state == SyncState.STOPPED
assert sync.stats.total_syncs == 0
@@ -205,11 +218,14 @@ class TestMarketMetadataSync:
assert sync._cache_ttl == DEFAULT_CACHE_TTL_SECONDS
assert sync._key_prefix == DEFAULT_REDIS_KEY_PREFIX
def test_init_custom_config(self, mock_redis: AsyncMock, mock_clob: MagicMock) -> None:
def test_init_custom_config(
self, mock_redis: AsyncMock, mock_clob: MagicMock, mock_gamma: MagicMock
) -> None:
"""Test initialization with custom config."""
sync = MarketMetadataSync(
redis=mock_redis,
clob_client=mock_clob,
gamma_client=mock_gamma,
sync_interval_seconds=60,
cache_ttl_seconds=120,
key_prefix="custom:",
@@ -220,9 +236,11 @@ class TestMarketMetadataSync:
assert sync._key_prefix == "custom:"
@pytest.mark.asyncio
async def test_start_stop(self, mock_redis: AsyncMock, mock_clob: MagicMock) -> None:
async def test_start_stop(
self, mock_redis: AsyncMock, mock_clob: MagicMock, mock_gamma: MagicMock
) -> None:
"""Test starting and stopping the sync service."""
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob)
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob, gamma_client=mock_gamma)
# Start
await sync.start()
@@ -236,10 +254,10 @@ class TestMarketMetadataSync:
@pytest.mark.asyncio
async def test_start_performs_initial_sync(
self, mock_redis: AsyncMock, mock_clob: MagicMock
self, mock_redis: AsyncMock, mock_clob: MagicMock, mock_gamma: MagicMock
) -> None:
"""Test that start performs an initial sync."""
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob)
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob, gamma_client=mock_gamma)
await sync.start()
@@ -252,10 +270,12 @@ class TestMarketMetadataSync:
await sync.stop()
@pytest.mark.asyncio
async def test_start_failure(self, mock_redis: AsyncMock, mock_clob: MagicMock) -> None:
async def test_start_failure(
self, mock_redis: AsyncMock, mock_clob: MagicMock, mock_gamma: MagicMock
) -> None:
"""Test start failure handling."""
mock_clob.get_markets.side_effect = Exception("API error")
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob)
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob, gamma_client=mock_gamma)
with pytest.raises(MetadataSyncError, match="initial sync failed"):
await sync.start()
@@ -268,6 +288,7 @@ class TestMarketMetadataSync:
self,
mock_redis: AsyncMock,
mock_clob: MagicMock,
mock_gamma: MagicMock,
sample_metadata: MarketMetadata,
) -> None:
"""Test get_market with cache hit."""
@@ -275,7 +296,7 @@ class TestMarketMetadataSync:
cached_data = json.dumps(sample_metadata.to_dict())
mock_redis.get = AsyncMock(return_value=cached_data)
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob)
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob, gamma_client=mock_gamma)
await sync.start()
result = await sync.get_market("cond123")
@@ -288,12 +309,14 @@ class TestMarketMetadataSync:
await sync.stop()
@pytest.mark.asyncio
async def test_get_market_cache_miss(self, mock_redis: AsyncMock, mock_clob: MagicMock) -> None:
async def test_get_market_cache_miss(
self, mock_redis: AsyncMock, mock_clob: MagicMock, mock_gamma: MagicMock
) -> None:
"""Test get_market with cache miss."""
# Setup cache miss
mock_redis.get = AsyncMock(return_value=None)
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob)
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob, gamma_client=mock_gamma)
await sync.start()
result = await sync.get_market("cond123")
@@ -308,12 +331,14 @@ class TestMarketMetadataSync:
await sync.stop()
@pytest.mark.asyncio
async def test_get_market_not_found(self, mock_redis: AsyncMock, mock_clob: MagicMock) -> None:
async def test_get_market_not_found(
self, mock_redis: AsyncMock, mock_clob: MagicMock, mock_gamma: MagicMock
) -> None:
"""Test get_market when market doesn't exist."""
mock_redis.get = AsyncMock(return_value=None)
mock_clob.get_market.return_value = None
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob)
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob, gamma_client=mock_gamma)
await sync.start()
result = await sync.get_market("nonexistent")
@@ -323,9 +348,11 @@ class TestMarketMetadataSync:
await sync.stop()
@pytest.mark.asyncio
async def test_invalidate_market(self, mock_redis: AsyncMock, mock_clob: MagicMock) -> None:
async def test_invalidate_market(
self, mock_redis: AsyncMock, mock_clob: MagicMock, mock_gamma: MagicMock
) -> None:
"""Test cache invalidation."""
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob)
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob, gamma_client=mock_gamma)
await sync.start()
result = await sync.invalidate_market("cond123")
@@ -336,9 +363,11 @@ class TestMarketMetadataSync:
await sync.stop()
@pytest.mark.asyncio
async def test_force_sync(self, mock_redis: AsyncMock, mock_clob: MagicMock) -> None:
async def test_force_sync(
self, mock_redis: AsyncMock, mock_clob: MagicMock, mock_gamma: MagicMock
) -> None:
"""Test forced sync."""
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob)
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob, gamma_client=mock_gamma)
await sync.start()
# Initial sync
@@ -353,7 +382,9 @@ class TestMarketMetadataSync:
await sync.stop()
@pytest.mark.asyncio
async def test_state_change_callback(self, mock_redis: AsyncMock, mock_clob: MagicMock) -> None:
async def test_state_change_callback(
self, mock_redis: AsyncMock, mock_clob: MagicMock, mock_gamma: MagicMock
) -> None:
"""Test state change callback."""
states: list[SyncState] = []
@@ -363,6 +394,7 @@ class TestMarketMetadataSync:
sync = MarketMetadataSync(
redis=mock_redis,
clob_client=mock_clob,
gamma_client=mock_gamma,
on_state_change=on_state_change,
)
@@ -377,7 +409,7 @@ class TestMarketMetadataSync:
@pytest.mark.asyncio
async def test_sync_complete_callback(
self, mock_redis: AsyncMock, mock_clob: MagicMock
self, mock_redis: AsyncMock, mock_clob: MagicMock, mock_gamma: MagicMock
) -> None:
"""Test sync complete callback."""
sync_stats: list[SyncStats] = []
@@ -388,6 +420,7 @@ class TestMarketMetadataSync:
sync = MarketMetadataSync(
redis=mock_redis,
clob_client=mock_clob,
gamma_client=mock_gamma,
on_sync_complete=on_sync_complete,
)
@@ -401,7 +434,11 @@ class TestMarketMetadataSync:
@pytest.mark.asyncio
async def test_get_markets_by_category(
self, mock_redis: AsyncMock, mock_clob: MagicMock, sample_metadata: MarketMetadata
self,
mock_redis: AsyncMock,
mock_clob: MagicMock,
mock_gamma: MagicMock,
sample_metadata: MarketMetadata,
) -> None:
"""Test getting markets by category."""
# Setup scan to return keys
@@ -412,7 +449,7 @@ class TestMarketMetadataSync:
cached_data = json.dumps(sample_metadata.to_dict())
mock_redis.get = AsyncMock(return_value=cached_data)
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob)
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob, gamma_client=mock_gamma)
# Don't start to avoid initial sync complexity
sync._state = SyncState.IDLE
@@ -422,9 +459,11 @@ class TestMarketMetadataSync:
assert results[0].category == "crypto"
@pytest.mark.asyncio
async def test_cannot_start_twice(self, mock_redis: AsyncMock, mock_clob: MagicMock) -> None:
async def test_cannot_start_twice(
self, mock_redis: AsyncMock, mock_clob: MagicMock, mock_gamma: MagicMock
) -> None:
"""Test that starting twice doesn't double-start."""
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob)
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob, gamma_client=mock_gamma)
await sync.start()
await sync.start() # Should be a no-op
@@ -434,9 +473,11 @@ class TestMarketMetadataSync:
await sync.stop()
@pytest.mark.asyncio
async def test_stop_when_stopped(self, mock_redis: AsyncMock, mock_clob: MagicMock) -> None:
async def test_stop_when_stopped(
self, mock_redis: AsyncMock, mock_clob: MagicMock, mock_gamma: MagicMock
) -> None:
"""Test stopping when already stopped."""
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob)
sync = MarketMetadataSync(redis=mock_redis, clob_client=mock_clob, gamma_client=mock_gamma)
await sync.stop() # Should be a no-op
+49 -14
View File
@@ -84,7 +84,10 @@ class TestTradeStreamHandler:
"""Test building subscription message without filters."""
msg = handler._build_subscription_message()
assert msg == {"subscriptions": [{"topic": "activity", "type": "trades"}]}
assert msg == {
"action": "subscribe",
"subscriptions": [{"topic": "activity", "type": "trades"}],
}
def test_build_subscription_message_with_event_filter(self, on_trade_mock: AsyncMock) -> None:
"""Test building subscription message with event filter."""
@@ -110,11 +113,10 @@ class TestTradeStreamHandler:
async def test_handle_message_trade(
self, handler: TradeStreamHandler, on_trade_mock: AsyncMock
) -> None:
"""Test handling a valid trade message."""
"""Test handling a valid trade message (payload-based routing)."""
message = json.dumps(
{
"topic": "activity",
"type": "trades",
"connection_id": "abc123",
"payload": {
"conditionId": "0xmarket",
"transactionHash": "0xtx",
@@ -142,11 +144,10 @@ class TestTradeStreamHandler:
async def test_handle_message_non_trade(
self, handler: TradeStreamHandler, on_trade_mock: AsyncMock
) -> None:
"""Test handling a non-trade message."""
"""Test handling a non-trade message (no transactionHash/proxyWallet)."""
message = json.dumps(
{
"topic": "comments",
"type": "comment_created",
"connection_id": "abc123",
"payload": {"body": "Hello"},
}
)
@@ -156,6 +157,35 @@ class TestTradeStreamHandler:
on_trade_mock.assert_not_called()
assert handler.stats.trades_received == 0
@pytest.mark.asyncio
async def test_handle_message_payload_missing_proxy_wallet(
self, handler: TradeStreamHandler, on_trade_mock: AsyncMock
) -> None:
"""Ratchet: payload with transactionHash but no proxyWallet is not a trade."""
message = json.dumps(
{
"connection_id": "abc",
"payload": {"transactionHash": "0xtx", "other": "field"},
}
)
await handler._handle_message(message)
on_trade_mock.assert_not_called()
assert handler.stats.trades_received == 0
@pytest.mark.asyncio
async def test_handle_message_no_payload_key(
self, handler: TradeStreamHandler, on_trade_mock: AsyncMock
) -> None:
"""Ratchet: message without payload key is ignored."""
message = json.dumps({"connection_id": "abc", "status": "ok"})
await handler._handle_message(message)
on_trade_mock.assert_not_called()
assert handler.stats.trades_received == 0
@pytest.mark.asyncio
async def test_handle_message_invalid_json(
self, handler: TradeStreamHandler, on_trade_mock: AsyncMock
@@ -175,8 +205,7 @@ class TestTradeStreamHandler:
message = json.dumps(
{
"topic": "activity",
"type": "trades",
"connection_id": "abc",
"payload": {
"conditionId": "0x",
"transactionHash": "0x",
@@ -231,14 +260,18 @@ class TestTradeStreamHandler:
mock_ws = AsyncMock()
mock_ws.send = AsyncMock()
with patch("websockets.connect", AsyncMock(return_value=mock_ws)):
with patch(
"polymarket_insider_tracker.ingestor.websocket.ws_connect",
AsyncMock(return_value=mock_ws),
):
ws = await handler._connect()
assert ws is mock_ws
mock_ws.send.assert_called_once()
# Verify subscription message
# Verify subscription message includes action: subscribe
sent_msg = json.loads(mock_ws.send.call_args[0][0])
assert sent_msg["action"] == "subscribe"
assert "subscriptions" in sent_msg
assert sent_msg["subscriptions"][0]["topic"] == "activity"
assert sent_msg["subscriptions"][0]["type"] == "trades"
@@ -289,8 +322,7 @@ class TestTradeStreamHandlerIntegration:
trade_message = json.dumps(
{
"topic": "activity",
"type": "trades",
"connection_id": "test-conn",
"payload": {
"conditionId": "0xtest",
"transactionHash": "0xtx",
@@ -333,7 +365,10 @@ class TestTradeStreamHandlerIntegration:
mock_ws = MockWebSocket(handler, trade_message)
with patch("websockets.connect", AsyncMock(return_value=mock_ws)):
with patch(
"polymarket_insider_tracker.ingestor.websocket.ws_connect",
AsyncMock(return_value=mock_ws),
):
# Run with timeout to prevent hanging
try:
await asyncio.wait_for(handler.start(), timeout=1.0)
+334
View File
@@ -0,0 +1,334 @@
"""Tests verifying wallet and funding data persistence in the pipeline.
These tests confirm that running the live pipeline writes rows into
wallet_profiles and funding_transfers tables when fresh wallets are detected.
"""
from __future__ import annotations
from datetime import UTC, datetime
from decimal import Decimal
from unittest.mock import AsyncMock, MagicMock
import pytest
from sqlalchemy import select
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from polymarket_insider_tracker.config import Settings
from polymarket_insider_tracker.detector.models import FreshWalletSignal
from polymarket_insider_tracker.ingestor.models import TradeEvent
from polymarket_insider_tracker.pipeline import Pipeline
from polymarket_insider_tracker.profiler.models import FundingChain, FundingTransfer, WalletProfile
from polymarket_insider_tracker.storage.database import DatabaseManager
from polymarket_insider_tracker.storage.models import Base, FundingTransferModel, WalletProfileModel
@pytest.fixture
def mock_settings():
"""Create mock settings for testing."""
redis = MagicMock()
redis.url = "redis://localhost:6379"
database = MagicMock()
database.url = "sqlite+aiosqlite:///:memory:"
polygon = MagicMock()
polygon.rpc_url = "https://polygon-rpc.com"
polygon.fallback_rpc_url = None
polymarket = MagicMock()
polymarket.ws_url = "wss://ws-subscriptions-clob.polymarket.com/ws/market"
polymarket.api_key = None
discord = MagicMock()
discord.enabled = False
discord.webhook_url = None
telegram = MagicMock()
telegram.enabled = False
telegram.bot_token = None
telegram.chat_id = None
settings = MagicMock(spec=Settings)
settings.redis = redis
settings.database = database
settings.polygon = polygon
settings.polymarket = polymarket
settings.discord = discord
settings.telegram = telegram
settings.dry_run = True
return settings
@pytest.fixture
async def async_engine():
"""Create an async SQLite engine for testing."""
engine = create_async_engine("sqlite+aiosqlite:///:memory:", echo=False)
async with engine.begin() as conn:
await conn.run_sync(Base.metadata.create_all)
yield engine
await engine.dispose()
@pytest.fixture
async def db_manager(async_engine):
"""Create a DatabaseManager backed by the in-memory SQLite engine."""
manager = DatabaseManager.__new__(DatabaseManager)
manager.database_url = "sqlite+aiosqlite:///:memory:"
manager.async_mode = True
manager._pool_size = 5
manager._max_overflow = 10
manager._echo = False
manager._sync_engine = None
manager._async_engine = async_engine
manager._sync_session_factory = None
manager._async_session_factory = async_sessionmaker(bind=async_engine, expire_on_commit=False)
return manager
@pytest.fixture
def sample_trade():
"""Create a sample trade event."""
return TradeEvent(
trade_id="0x" + "a" * 64,
wallet_address="0x" + "b" * 40,
market_id="0x" + "c" * 64,
asset_id="asset_123",
side="BUY",
price=Decimal("0.65"),
size=Decimal("5000"),
timestamp=datetime.now(UTC),
outcome="Yes",
outcome_index=0,
event_title="Test Market",
market_slug="test-market",
)
@pytest.fixture
def sample_profile():
"""Create a sample fresh wallet profile."""
return WalletProfile(
address="0x" + "b" * 40,
nonce=2,
first_seen=datetime(2026, 3, 31, 12, 0, 0, tzinfo=UTC),
age_hours=1.5,
is_fresh=True,
total_tx_count=2,
matic_balance=Decimal("1000000000000000000"),
usdc_balance=Decimal("5000000000"),
fresh_threshold=5,
)
@pytest.fixture
def sample_funding_chain():
"""Create a sample funding chain with one transfer."""
return FundingChain(
target_address="0x" + "b" * 40,
chain=[
FundingTransfer(
from_address="0x" + "d" * 40,
to_address="0x" + "b" * 40,
amount=Decimal("5000000000"),
token="USDC",
tx_hash="0x" + "e" * 64,
block_number=12345678,
timestamp=datetime(2026, 3, 31, 11, 0, 0, tzinfo=UTC),
),
],
origin_address="0x" + "d" * 40,
origin_type="cex_binance",
hop_count=1,
)
class TestPipelinePersistence:
"""Tests that the pipeline persists wallet and funding data to Postgres."""
@pytest.mark.asyncio
async def test_on_trade_persists_wallet_profile(
self, mock_settings, db_manager, sample_trade, sample_profile, async_engine
):
"""When a fresh wallet signal fires, the wallet profile is written to wallet_profiles."""
pipeline = Pipeline(mock_settings)
pipeline._db_manager = db_manager
fresh_signal = FreshWalletSignal(
trade_event=sample_trade,
wallet_profile=sample_profile,
confidence=0.8,
factors={"base": 0.5, "brand_new": 0.2},
)
pipeline._fresh_wallet_detector = MagicMock()
pipeline._fresh_wallet_detector.analyze = AsyncMock(return_value=fresh_signal)
pipeline._size_anomaly_detector = MagicMock()
pipeline._size_anomaly_detector.analyze = AsyncMock(return_value=None)
pipeline._funding_tracer = MagicMock()
pipeline._funding_tracer.trace = AsyncMock(
return_value=FundingChain(target_address=sample_profile.address)
)
pipeline._risk_scorer = MagicMock()
pipeline._risk_scorer.assess = AsyncMock(
return_value=MagicMock(should_alert=False, weighted_score=0.3)
)
pipeline._alert_formatter = MagicMock()
pipeline._alert_dispatcher = MagicMock()
await pipeline._on_trade(sample_trade)
# Verify wallet_profiles has a row
async with async_sessionmaker(bind=async_engine, expire_on_commit=False)() as session:
result = await session.execute(select(WalletProfileModel))
rows = result.scalars().all()
assert len(rows) == 1
assert rows[0].address == sample_profile.address.lower()
assert rows[0].nonce == sample_profile.nonce
assert rows[0].is_fresh is True
@pytest.mark.asyncio
async def test_on_trade_persists_funding_transfers(
self,
mock_settings,
db_manager,
sample_trade,
sample_profile,
sample_funding_chain,
async_engine,
):
"""When a fresh wallet signal fires, funding transfers are written to funding_transfers."""
pipeline = Pipeline(mock_settings)
pipeline._db_manager = db_manager
fresh_signal = FreshWalletSignal(
trade_event=sample_trade,
wallet_profile=sample_profile,
confidence=0.8,
factors={"base": 0.5, "brand_new": 0.2},
)
pipeline._fresh_wallet_detector = MagicMock()
pipeline._fresh_wallet_detector.analyze = AsyncMock(return_value=fresh_signal)
pipeline._size_anomaly_detector = MagicMock()
pipeline._size_anomaly_detector.analyze = AsyncMock(return_value=None)
pipeline._funding_tracer = MagicMock()
pipeline._funding_tracer.trace = AsyncMock(return_value=sample_funding_chain)
pipeline._risk_scorer = MagicMock()
pipeline._risk_scorer.assess = AsyncMock(
return_value=MagicMock(should_alert=False, weighted_score=0.3)
)
pipeline._alert_formatter = MagicMock()
pipeline._alert_dispatcher = MagicMock()
await pipeline._on_trade(sample_trade)
# Verify funding_transfers has a row
async with async_sessionmaker(bind=async_engine, expire_on_commit=False)() as session:
result = await session.execute(select(FundingTransferModel))
rows = result.scalars().all()
assert len(rows) == 1
assert rows[0].to_address == ("0x" + "b" * 40).lower()
assert rows[0].from_address == ("0x" + "d" * 40).lower()
assert rows[0].token == "USDC"
assert rows[0].tx_hash == ("0x" + "e" * 64).lower()
@pytest.mark.asyncio
async def test_no_persistence_without_fresh_signal(
self, mock_settings, db_manager, sample_trade, async_engine
):
"""No rows written when fresh wallet signal is None (wallet not fresh)."""
pipeline = Pipeline(mock_settings)
pipeline._db_manager = db_manager
pipeline._fresh_wallet_detector = MagicMock()
pipeline._fresh_wallet_detector.analyze = AsyncMock(return_value=None)
pipeline._size_anomaly_detector = MagicMock()
pipeline._size_anomaly_detector.analyze = AsyncMock(return_value=None)
await pipeline._on_trade(sample_trade)
async with async_sessionmaker(bind=async_engine, expire_on_commit=False)() as session:
wallets = (await session.execute(select(WalletProfileModel))).scalars().all()
transfers = (await session.execute(select(FundingTransferModel))).scalars().all()
assert len(wallets) == 0
assert len(transfers) == 0
@pytest.mark.asyncio
async def test_persistence_failure_does_not_break_pipeline(
self, mock_settings, sample_trade, sample_profile
):
"""Persistence errors are caught and don't crash trade processing."""
pipeline = Pipeline(mock_settings)
# Use a broken db_manager that raises on get_async_session
broken_db = MagicMock()
broken_db.get_async_session = MagicMock(side_effect=Exception("DB connection failed"))
pipeline._db_manager = broken_db
fresh_signal = FreshWalletSignal(
trade_event=sample_trade,
wallet_profile=sample_profile,
confidence=0.8,
factors={"base": 0.5},
)
pipeline._fresh_wallet_detector = MagicMock()
pipeline._fresh_wallet_detector.analyze = AsyncMock(return_value=fresh_signal)
pipeline._size_anomaly_detector = MagicMock()
pipeline._size_anomaly_detector.analyze = AsyncMock(return_value=None)
pipeline._funding_tracer = MagicMock()
pipeline._risk_scorer = MagicMock()
pipeline._risk_scorer.assess = AsyncMock(
return_value=MagicMock(should_alert=False, weighted_score=0.3)
)
pipeline._alert_formatter = MagicMock()
pipeline._alert_dispatcher = MagicMock()
# Should not raise
await pipeline._on_trade(sample_trade)
assert pipeline.stats.trades_processed == 1
@pytest.mark.asyncio
async def test_duplicate_funding_transfers_are_skipped(
self,
mock_settings,
db_manager,
sample_trade,
sample_profile,
sample_funding_chain,
async_engine,
):
"""Processing the same trade twice should not duplicate funding transfer rows."""
pipeline = Pipeline(mock_settings)
pipeline._db_manager = db_manager
fresh_signal = FreshWalletSignal(
trade_event=sample_trade,
wallet_profile=sample_profile,
confidence=0.8,
factors={"base": 0.5},
)
pipeline._fresh_wallet_detector = MagicMock()
pipeline._fresh_wallet_detector.analyze = AsyncMock(return_value=fresh_signal)
pipeline._size_anomaly_detector = MagicMock()
pipeline._size_anomaly_detector.analyze = AsyncMock(return_value=None)
pipeline._funding_tracer = MagicMock()
pipeline._funding_tracer.trace = AsyncMock(return_value=sample_funding_chain)
pipeline._risk_scorer = MagicMock()
pipeline._risk_scorer.assess = AsyncMock(
return_value=MagicMock(should_alert=False, weighted_score=0.3)
)
pipeline._alert_formatter = MagicMock()
pipeline._alert_dispatcher = MagicMock()
# Process same trade twice
await pipeline._on_trade(sample_trade)
await pipeline._on_trade(sample_trade)
# Should still have only 1 funding transfer (duplicate skipped)
async with async_sessionmaker(bind=async_engine, expire_on_commit=False)() as session:
result = await session.execute(select(FundingTransferModel))
rows = result.scalars().all()
assert len(rows) == 1