The size anomaly detector currently emits a base 0.2 confidence signal
whenever a market is "niche" (low volume OR niche-prone category with
unknown volume) — even for $5 trades. In production this floods the
alert pipeline with nothing-burgers, because the niche-prone category
set covers `science / tech / finance / other`, which matches a huge
chunk of Polymarket's long tail.
This adds a `DEFAULT_NICHE_MIN_TRADE_SIZE = $500` floor that applies
ONLY to the niche-only path: if a trade exceeds the volume or book
thresholds, the guard does not block it. So real anomalies still come
through, but tiny niche trades get filtered out before reaching the
risk scorer.
The threshold is configurable via `niche_min_trade_size` in
`SizeAnomalyDetector.__init__`.
Tests added:
- niche-only below floor → suppressed
- niche-only at/above floor → still emits 0.2 base
- small trade with high volume_impact → guard does not block
Co-authored-by: schrodinger01 <schrodinger01@users.noreply.github.com>
* feat(detector): persist all risk assessments to risk_assessments table
* docs+test: add CHANGELOG and persistence regression tests
Documents the persist-all-assessments feature shipped in 8a0e8c9 and adds two regression tests covering: (1) sub-threshold assessments still hit the DB, and (2) DB write failures do not block alert dispatch.
* fix: add detector mock to pipeline test fixture
The persist_assessments feature accesses settings.detector which the
existing mock_settings fixture didn't include.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: ruff lint and format fixes for persist assessment
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: jp-vps-deploy <vps-deploy@schrodinger01>
Co-authored-by: schrodinger01 <schrodinger01@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(profiler): chunk eth_getLogs into <=10k-block windows
Public Polygon RPC providers (publicnode, ankr, llamarpc) cap eth_getLogs
at 10_000 blocks per request. The funding tracer was calling get_logs
with from_block=0 / to_block="latest", so every funding chain trace
failed in production with:
{'code': -32701, 'message': 'exceed maximum block range: 10000'}
Resolve the symbolic range to concrete bounds (default lookback ~30 days
of Polygon blocks) and walk the window in 9_000-block chunks, oldest
first, stopping early once `limit` matches are gathered. Walking
oldest-first preserves the "first transfer" semantics the funding tracer
already relies on.
Includes 4 new tests:
- chunks_large_ranges: regression guard that no single window exceeds
the cap
- stops_when_limit_hit_mid_walk: short-circuits once enough hits
- skips_failing_chunk: a flaky window doesn't tank the whole trace
- resolves_latest_via_block_number: from_block=0 + to_block="latest"
resolves to the last max_lookback_blocks
The 3 pre-existing _get_transfer_logs tests now pass explicit numeric
ranges so they don't go through the latest-resolution path; coverage of
that path is moved to the new dedicated test.
* fix(profiler): cap default lookback at 80k blocks + early-break on pruned
Field-test of the chunking fix on a public Polygon RPC (publicnode)
revealed a second wall behind the first: after a chunk request lands
outside the provider's archive horizon, every subsequent chunk fails
with the same error:
{'code': -32701, 'message': 'History has been pruned for this block.
To remove restrictions, order a dedicated full node here: ...'}
publicnode empirically retains roughly the most recent 100_000 blocks
(~55 hours) of log history. Surveying other public free-tier RPCs:
drpc.org — archive, but rejects ranges >= ~1_000 blocks
llamarpc — empty responses on archive ranges
ankr — now requires API key
blockpi/onfin — block-range limits 50–500
1rpc.io/matic — limited to 50 blocks
Two changes to make funding traces actually return data on a public
RPC instead of swallowing 140 pruned-history warnings per wallet:
1. Lower DEFAULT_MAX_LOOKBACK_BLOCKS from 1_300_000 to 80_000. Fresh
wallets — the population this signal exists to flag — are by
definition new, so a ~44 hour window covers their entire funding
history. Older wallets lose archive coverage on free RPCs but
they're not what the fresh-wallet signal scores on anyway.
2. Detect pruned-history errors by message substring and short-circuit
the chunk walk. Walking further back is futile once we're past the
cutoff; bailing early avoids burning RPC quota on chunks that are
guaranteed to fail.
Both knobs remain constructor parameters — deployments behind a paid
archive node can dial DEFAULT_MAX_LOOKBACK_BLOCKS back up.
Two new tests:
- test_get_transfer_logs_breaks_on_pruned_history: pruned error on
chunk #2 must keep chunk #3 from ever being issued
- test_get_transfer_logs_default_lookback_fits_pruned_horizon:
regression guard pinning the default at <= 100_000 so a future
refactor doesn't silently re-introduce the unusable default
* fix(profiler): 0x-prefix the Transfer event topic for strict RPC providers
`HexBytes.hex()` returns a bare hex string with no `0x` prefix. publicnode
tolerates that, but drpc — which we use as the failover RPC — rejects it
outright with `invalid argument 0: hex string without 0x prefix`, and every
single eth_getLogs chunk in the funding trace fails. Once the primary is
flipped to unhealthy by any other call, the entire funding subsystem
silently produces zero rows in funding_transfers.
Switch to a precomputed `TRANSFER_EVENT_TOPIC` constant that always carries
the `0x` prefix, and add a regression test that asserts the topic shape
sent to eth_getLogs.
* fix: lint/format fixes for eth_getLogs chunking
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: schrodinger01 <schrodinger01@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The CLOB API does not expose 24h volume or order-book liquidity, so the
size_anomaly detector currently has no real ratio to compare a trade
against and falls back to the niche-base 0.2 confidence floor. That makes
the volume_impact / book_impact thresholds essentially dead code.
This change adds a small client for the public gamma-api markets endpoint
and merges its volume24hr / liquidityNum snapshot into MarketMetadata
during the existing periodic sync. The detector can now compute real
volume and book impact ratios.
Notes on the gamma client:
- gamma-api enforces a server-side max of 100 markets per page and caps
`offset` around 10000. The client paginates with bounded concurrency,
sorted by `volume24hr desc`, so the most-traded markets (which is where
size anomalies actually matter) are always covered. Markets beyond that
window have negligible recent volume and the niche path handles them
fine without a ratio.
- Failures are swallowed: a degraded gamma endpoint must not stop CLOB
metadata from being cached, since size_anomaly + the niche path remain
functional with `daily_volume=None`.
MarketMetadata gains three optional Decimal fields (daily_volume,
weekly_volume, liquidity); to_dict / from_dict round-trip is preserved
and older cache entries without these keys deserialize cleanly.
Tests: 12 new tests for GammaClient (parsing, single-page, short-page
stop, offset-cap clean stop, retry, malformed responses); existing
metadata_sync tests updated to inject a mocked GammaClient so they don't
hit the real network.
Co-authored-by: schrodinger01 <schrodinger01@users.noreply.github.com>
Telegram's MarkdownV2 parser treats `.` as a special character and rejects
the message with `Bad Request: can't parse entities` if any unescaped `.`
appears outside a code or pre block. The current formatter only escapes
the static `Market:` title and the `$` in the USDC amount, while leaving
three numeric runs unescaped:
*Risk Score:* 0.82 (HIGH) ← `.` in the score literal
*Trade:* BUY Yes @ $0.075 | $15,000.00
↑ ↑
price USDC amount
In production this means well-formed alerts silently fail to reach
Telegram users — the dispatcher reports a 400 from the Bot API and the
event is dropped.
This change routes every dynamic value through `_escape_telegram_markdown`
before interpolation:
- `assessment.weighted_score` (risk score)
- `trade.price` (price string)
- `format_usdc(trade.notional_value)` (USDC amount, including its `.`)
- `trade.side` and `trade.outcome` (defensive — upstream values may
contain `-` or `.` in future schema changes)
The test that previously asserted `"0.82" in result.telegram_markdown` is
updated to require the escaped form `"0\.82"`, and a new test pins down
that none of `0.82` / `0.075` / `15,000.00` appear in unescaped form.
Co-authored-by: schrodinger01 <schrodinger01@users.noreply.github.com>
* fix(ingestor): align WebSocket subscribe + routing with live API
The Polymarket ws-live-data WebSocket requires `action: "subscribe"` in
the subscribe envelope. Without it the server accepts the connection but
never delivers trade events, causing the tracker to silently produce
zero alerts.
Additionally, incoming frames are shaped `{connection_id, payload:{...}}`
-- they do NOT echo the `topic`/`type` keys we sent. The previous routing
check matched nothing and every real trade was silently dropped.
Changes:
- Add `action: "subscribe"` to subscription message
- Route incoming messages by payload shape (transactionHash + proxyWallet)
- Add ratchet tests for payload routing edge cases
- Rewrite README as agent-first with <2min quickstart
- Add skill draft (docs/skill-tracking-prediction-market-flow.md)
Closes#89
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(lint): remove unused imports in test_pipeline_persistence
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* style: apply ruff formatting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The live pipeline was receiving trade data and populating Redis caches but
never persisting wallet profiles or funding transfers to Postgres. This
wires the repositories into the _on_trade flow: when a fresh wallet signal
is detected, the wallet profile is upserted to wallet_profiles and the
funding chain is traced and inserted into funding_transfers. Existing
Redis caching behavior is preserved.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: replace deprecated datetime.utcnow() and websockets.legacy APIs
- Replace datetime.utcnow() with datetime.now(UTC) in Orderbook model
- Use websockets.asyncio.client.connect instead of legacy websockets.connect
- Import ConnectionClosed from websockets.exceptions directly
- Update test mocks to patch the new import paths
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve ruff lint errors in alembic migration
- Replace typing.Union with X | Y syntax (UP007)
- Import Sequence from collections.abc instead of typing (UP035)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: remove deprecated version key from docker-compose.yml
The top-level 'version' key is obsolete in modern Docker Compose
and produces a warning.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: fix README to match actual project structure and tooling
- Replace pip commands with uv equivalents
- Fix run command from 'python -m src.main' to 'python -m polymarket_insider_tracker'
- Update project structure tree to reflect actual src/polymarket_insider_tracker/ layout
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
StrEnum (Python 3.11+) is the modern replacement for the (str, Enum)
pattern. Updates SyncState and PipelineState enums.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
.env files don't support shell variable expansion, so DATABASE_URL and
REDIS_URL contained literal ${...} strings. Also alembic/env.py read
SQLALCHEMY_DATABASE_URL instead of DATABASE_URL.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Replace shell variable interpolation in .env.example with literal
values since .env files don't support variable expansion
- Change alembic/env.py to read DATABASE_URL (matching app convention)
instead of SQLALCHEMY_DATABASE_URL
Closes#62
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Replace shell variable interpolation in .env.example with literal values
- Change alembic/env.py to read DATABASE_URL instead of SQLALCHEMY_DATABASE_URL
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Replace shell variable interpolation in .env.example with literal values
- Change alembic/env.py to read DATABASE_URL instead of SQLALCHEMY_DATABASE_URL
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Replace shell variable interpolation in .env.example with literal
values since .env files don't expand variables
- Change alembic/env.py to read DATABASE_URL instead of
SQLALCHEMY_DATABASE_URL to match the app's env var name
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
.env files don't support shell variable interpolation, so DATABASE_URL
and REDIS_URL had literal ${VAR} strings instead of actual values.
Also alembic/env.py read SQLALCHEMY_DATABASE_URL instead of DATABASE_URL.
Closes#62
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Replace shell variable interpolation in .env.example with literal
default values (.env files don't expand variables)
- Change alembic/env.py to read DATABASE_URL instead of
SQLALCHEMY_DATABASE_URL to match the app's convention
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Replace shell variable interpolation in .env.example with literal
default values since .env files don't expand variables
- Change alembic/env.py to read DATABASE_URL instead of
SQLALCHEMY_DATABASE_URL to match the app's env var name
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Replace shell variable interpolation in .env.example with literal
values (.env files don't expand variables)
- Change alembic/env.py to read DATABASE_URL instead of
SQLALCHEMY_DATABASE_URL to match app configuration
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Replace shell variable interpolation in .env.example with literal
values (.env files don't expand variables)
- Change alembic/env.py to read DATABASE_URL instead of
SQLALCHEMY_DATABASE_URL to match app convention
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
.env files don't support shell variable interpolation, so DATABASE_URL
and REDIS_URL had literal ${VAR} strings instead of actual values.
Also alembic/env.py was reading SQLALCHEMY_DATABASE_URL instead of
DATABASE_URL, which is what the app and .env use.
Closes#62
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Replace shell variable interpolation in .env.example with literal
values (.env files don't expand ${VAR} syntax)
- Change alembic/env.py to read DATABASE_URL instead of
SQLALCHEMY_DATABASE_URL to match the app's env var name
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Replace shell variable interpolation in .env.example with literal
values (.env files don't expand ${VAR} syntax)
- Change alembic/env.py to read DATABASE_URL instead of
SQLALCHEMY_DATABASE_URL to match the app's env var name
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Replace shell variable interpolation in .env.example with literal values
- Fix alembic/env.py to read DATABASE_URL instead of SQLALCHEMY_DATABASE_URL
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1. Replace shell variable interpolation in .env.example with literal
values, since .env files don't expand variables.
2. Change alembic/env.py to read DATABASE_URL instead of
SQLALCHEMY_DATABASE_URL to match app convention.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1. Replace shell variable interpolation in .env.example with literal
values, since .env files don't expand variables.
2. Change alembic/env.py to read DATABASE_URL instead of
SQLALCHEMY_DATABASE_URL to match app convention.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1. Replace shell variable interpolation in .env.example with literal
values, since .env files don't expand variables.
2. Change alembic/env.py to read DATABASE_URL instead of
SQLALCHEMY_DATABASE_URL to match app convention.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Replace shell variable interpolation in .env.example with literal values
- Change alembic/env.py to read DATABASE_URL instead of SQLALCHEMY_DATABASE_URL
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Replace shell variable interpolation in .env.example with literal
default values (.env files don't expand variables)
- Change alembic/env.py to read DATABASE_URL instead of
SQLALCHEMY_DATABASE_URL to match the app's env var name
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Replace shell variable interpolation in .env.example with literal
values, since .env files don't expand shell variables
- Fix alembic/env.py to read DATABASE_URL instead of SQLALCHEMY_DATABASE_URL
to match the app's environment variable convention
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Implement graceful shutdown handling for the async pipeline with proper
signal trapping and cleanup coordination.
Features:
- GracefulShutdown class with SIGTERM/SIGINT signal trapping
- Async event-based shutdown coordination
- Configurable shutdown timeout (default: 30 seconds)
- Cleanup callback registration (sync and async)
- Force exit on second signal
- Async context manager support
- Platform-specific signal handling (Unix/Windows)
Integration:
- Updated __main__.py to use GracefulShutdown wrapper
- Pipeline cleanup registered as shutdown callback
- Proper logging of shutdown stages
Tests: 25 new tests covering signal handling, timeouts, cleanup callbacks,
and context manager behavior.
Closes#56🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement the main CLI entry point for the Polymarket Insider Tracker.
Features:
- argparse-based CLI with version, config-check, dry-run, log-level flags
- Structured logging configuration with multiple formatters
- Configuration validation with friendly error messages
- Startup banner display
- Config check mode for validation without running
- Integration with Pipeline orchestrator
Tests: 20 new tests covering parser, logging, banner, config validation,
and integration scenarios.
Closes#55🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add Pipeline class that wires together all detection components and
manages the event flow from trade ingestion to alerting.
Key features:
- Lifecycle management (start/stop) with proper cleanup
- Component initialization from Settings
- Parallel detector execution for fresh wallet and size anomaly
- Risk scoring with alert threshold
- Multi-channel alert dispatch (Discord/Telegram)
- Dry-run mode for testing without sending alerts
- Async context manager support
- Comprehensive test coverage (19 tests)
Pipeline flow: WebSocket → Profiler → Detectors → Scorer → Alerter
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add pydantic-settings dependency
- Create centralized config.py with nested settings groups:
- DatabaseSettings: PostgreSQL connection validation
- RedisSettings: Redis connection with defaults
- PolygonSettings: RPC endpoints with fallback support
- PolymarketSettings: WebSocket URL and optional API key
- DiscordSettings: Optional webhook for alerts
- TelegramSettings: Optional bot token/chat ID for alerts
- Implement singleton pattern with get_settings() and LRU cache
- Add redacted_summary() for logging config without exposing secrets
- Support .env file loading via python-dotenv
- Add comprehensive test suite (26 tests)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
## Changes
### test_get_market_not_found
- Updated test to expect `RetryError` instead of `ClobClientError`
- The `get_market` method uses the `@with_retry()` decorator, so when
the underlying API call fails repeatedly, it raises `RetryError`
wrapping the original exception
- Removed unused `ClobClientError` import
### test_start_and_receive_trades
- Replaced `MagicMock` with a proper `MockWebSocket` class that
implements the async iterator protocol correctly
- `MagicMock` was passing `self` as an extra argument when calling
`__aiter__`, causing "takes 0 positional arguments but 1 was given"
- Removed unused `MagicMock` import
Closes#49🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Revert underscore-prefixed fixture parameters (breaks pytest discovery)
- Use # noqa: ARG002 comments instead
- Add aiosqlite to dev dependencies for SQLite-based tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Patrick Selamy