diff --git a/.env.example b/.env.example index aaf6c92..7763656 100644 --- a/.env.example +++ b/.env.example @@ -27,6 +27,10 @@ RECORD_FEED=false # Observability — https://sentry.io (project: arbpulse) SENTRY_DSN= +# Tracing/spans gate — default off. Error monitoring is always on when SENTRY_DSN +# is set; spans are only created/exported when SENTRY_TRACING is enabled. Kept off +# by default so 24/7 operation stays within Sentry's free-tier span quota. +SENTRY_TRACING=false # Structured logs (pino); correlation IDs on REST, SSE, and WS hot path LOG_LEVEL=info diff --git a/README.md b/README.md index 71d194f..3bc5a3d 100644 --- a/README.md +++ b/README.md @@ -347,7 +347,7 @@ Eso no indica que el bot “no funcione”: indica que el filtro económico es e - **Pre-commit:** Husky runs lint-staged (`eslint --fix`, `prettier --write`) on staged `*.ts` / `*.tsx` in `src/` and `web/src/`. - **API contracts:** Zod schemas per REST endpoint → OpenAPI via `@asteasolutions/zod-to-openapi` → Scalar UI at `/api-docs`. Request bodies validated with Zod in route handlers. -- **Observability:** `@sentry/node` captures REST, WebSocket feed, and SSE errors; `pino` JSON logs with `correlationId` (from `x-request-id` or per book tick); OpenTelemetry spans (`ws.message` → `orderbook.process` → `arbitrage.evaluate` → `sse.broadcast`) export to Sentry via `@sentry/opentelemetry` when `SENTRY_DSN` is set. Dev probe: `GET /api/debug/sentry`; verify with `npm run test:observability`. +- **Observability:** `@sentry/node` captures REST, WebSocket feed, and SSE errors (always on when `SENTRY_DSN` is set); `pino` JSON logs with `correlationId` (from `x-request-id` or per book tick); OpenTelemetry spans (`ws.message` → `orderbook.process` → `arbitrage.evaluate` → `sse.broadcast`) export to Sentry via `@sentry/opentelemetry` **only when `SENTRY_TRACING` is enabled** (default off, so 24/7 operation stays within the free-tier span quota). With tracing off the tracer is a no-op — no spans are created or exported — but errors are still captured. Dev probe: `GET /api/debug/sentry`; verify with `npm run test:observability`. - **Rate limiting & cache:** Upstash sliding-window limits per IP on read/write/SSE routes (`429` + `Retry-After`). Latest `StateSnapshot` cached in Redis (~1s TTL) for `GET /api/state` and refreshed on SSE broadcast. Set `UPSTASH_REDIS_REST_URL` and `UPSTASH_REDIS_REST_TOKEN`. Verify with `npm run test:rate-limit`. - **CI:** GitHub Actions quality pipeline (typecheck, unit tests, build); Playwright smoke on PRs — dashboard loads and SSE connects (`npm run test:e2e`, workflow `e2e.yml`, `DEMO_MODE=true` in CI). - **Security scanning:** CodeQL (`.github/workflows/codeql.yml`); Dependabot for npm (root + `web/`) and GitHub Actions. diff --git a/openspec/changes/gate-sentry-spans/.openspec.yaml b/openspec/changes/gate-sentry-spans/.openspec.yaml new file mode 100644 index 0000000..cd2ce7e --- /dev/null +++ b/openspec/changes/gate-sentry-spans/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-07-16 diff --git a/openspec/changes/gate-sentry-spans/design.md b/openspec/changes/gate-sentry-spans/design.md new file mode 100644 index 0000000..25b4b0e --- /dev/null +++ b/openspec/changes/gate-sentry-spans/design.md @@ -0,0 +1,34 @@ +## Context + +`src/instrumentation/otel.ts` builds a tracer that, when `SENTRY_DSN` is set, registers a `NodeTracerProvider` with `SentrySpanProcessor`/`SentryPropagator`, so every `withSpan(...)` call on the hot path creates and exports a span to Sentry. `src/instrumentation/sentry.ts` sets `tracesSampleRate` unconditionally. Running 24/7 on a VPS multiplies span volume against Sentry's free-tier quota (5M spans/month), while error events stay small. We want tracing to remain a debugging option but be off by default. + +## Goals / Non-Goals + +**Goals:** +- Add a single boolean env var (`SENTRY_TRACING`, default `false`) that turns span creation/export on or off. +- Default behavior emits zero spans and zero hot-path span allocation. +- Preserve error monitoring exactly as-is regardless of the flag. +- No dependency changes; no API/wire changes. + +**Non-Goals:** +- Removing OpenTelemetry or the `withSpan` abstraction. +- Making `tracesSampleRate` itself env-configurable (kept as current 0.1 prod / 1 dev when enabled). +- Changing which operations are wrapped. + +## Decisions + +- **Boolean gate over sample-rate gate.** A boolean `SENTRY_TRACING` that makes the tracer a no-op avoids hot-path span allocation entirely when off (aligns with the "no unnecessary alloc on the hot path" convention). A `tracesSampleRate=0` approach would still register the provider and create spans that are dropped later — more overhead. Alternative considered and rejected. +- **Gate lives in `otel.ts` init.** `initOpenTelemetry()` returns the no-op `trace.getTracer("arbpulse")` unless both `SENTRY_DSN` and `SENTRY_TRACING` are truthy. This keeps `withSpan`'s signature and all call sites unchanged; its `catch` still calls `Sentry.captureException`, so errors are captured even with a no-op span. +- **Conditional `tracesSampleRate` in `sentry.ts`.** Only set `tracesSampleRate` when tracing is enabled; omit it otherwise so Sentry performance is fully off. A shared helper `isTracingEnabled()` reads the flag so both files agree. +- **Parsing.** Reuse the existing truthy convention (`"1"` or `"true"`, case-insensitive) used by `config.ts`'s `bool()` for consistency; implement locally in the instrumentation module since it initializes before `config` import. + +## Risks / Trade-offs + +- [Someone enables tracing in prod and hits the quota] → Documented in `.env.example`/README that it is off by default and why; enabling is an explicit opt-in. +- [Flag read in two files could drift] → Centralize in one `isTracingEnabled()` helper exported from the instrumentation layer. +- [No-op tracer still wraps hot path in a promise] → Accepted: `withSpan` remains async as today; the change only removes span export, matching prior behavior when `SENTRY_DSN` was unset. + +## Migration Plan + +- Deploy with `SENTRY_TRACING` unset → tracing off (safe default), errors still reported. +- To debug, set `SENTRY_TRACING=true` (with `SENTRY_DSN`) temporarily; unset to roll back. No data migration. diff --git a/openspec/changes/gate-sentry-spans/proposal.md b/openspec/changes/gate-sentry-spans/proposal.md new file mode 100644 index 0000000..07e7aed --- /dev/null +++ b/openspec/changes/gate-sentry-spans/proposal.md @@ -0,0 +1,25 @@ +## Why + +Arb Pulse is moving to a 24/7 VPS (Hetzner), where the engine runs continuously and emits OpenTelemetry spans on the hot path (`ws.message` → `orderbook.process` → `arbitrage.evaluate` → `sse.broadcast`). At that volume the app can approach or exceed Sentry's free-tier span quota (5M spans/month), while error monitoring stays well within limits. We want to keep tracing available for debugging but off by default so continuous operation is free-tier safe. + +## What Changes + +- Introduce an environment variable (`SENTRY_TRACING`, boolean, default `false`) that gates OpenTelemetry span creation and export to Sentry. +- When `SENTRY_TRACING` is disabled (default), the tracer is a no-op: no spans are created or exported, and no hot-path span allocation occurs. Error capture (`Sentry.captureException`, `unhandledRejection`, `uncaughtException`) remains fully active. +- When `SENTRY_TRACING` is enabled **and** `SENTRY_DSN` is set, spans are created and exported to Sentry as today, and `tracesSampleRate` is applied. +- Document the new variable in `.env.example` and `README.md`. + +## Capabilities + +### New Capabilities +- `observability`: error monitoring and optional distributed tracing behavior — how Sentry error capture is always on, and how OpenTelemetry span emission is conditioned on an environment flag. + +### Modified Capabilities + + +## Impact + +- Code: `src/instrumentation/otel.ts` (gate span provider/export), `src/instrumentation/sentry.ts` (conditional `tracesSampleRate`), possibly `src/instrumentation/index.ts`. +- Config/docs: `.env.example`, `README.md` (observability section + env var table). +- Dependencies: none added or removed (OTel packages stay for the enabled path). +- Behavior: default runtime emits zero spans; error monitoring unchanged. No API or wire-format changes. diff --git a/openspec/changes/gate-sentry-spans/specs/observability/spec.md b/openspec/changes/gate-sentry-spans/specs/observability/spec.md new file mode 100644 index 0000000..db1245f --- /dev/null +++ b/openspec/changes/gate-sentry-spans/specs/observability/spec.md @@ -0,0 +1,43 @@ +## ADDED Requirements + +### Requirement: Error monitoring is always active + +The system SHALL report runtime errors to Sentry whenever `SENTRY_DSN` is configured, independent of any tracing configuration. This includes REST, WebSocket feed, and SSE errors captured via `Sentry.captureException`, plus process-level `unhandledRejection` and `uncaughtException` handlers. + +#### Scenario: Error captured with DSN set + +- **WHEN** `SENTRY_DSN` is set and a handled error occurs on the WebSocket, REST, or SSE path +- **THEN** the error is sent to Sentry as an error event + +#### Scenario: No DSN configured + +- **WHEN** `SENTRY_DSN` is empty or unset +- **THEN** Sentry is not initialized and no events are sent, and the application continues running normally + +### Requirement: Tracing spans are gated by an environment flag + +The system SHALL only create and export OpenTelemetry spans when the `SENTRY_TRACING` environment variable is enabled AND `SENTRY_DSN` is set. The flag SHALL default to disabled so that continuous 24/7 operation emits zero spans by default. + +#### Scenario: Tracing disabled by default + +- **WHEN** `SENTRY_TRACING` is unset or falsy (default) +- **THEN** the tracer is a no-op, no spans are created on the hot path, and no spans are exported to Sentry + +#### Scenario: Tracing enabled with DSN + +- **WHEN** `SENTRY_TRACING` is truthy and `SENTRY_DSN` is set +- **THEN** hot-path spans (`ws.message`, `orderbook.process`, `arbitrage.evaluate`, `sse.broadcast`) are created and exported to Sentry, and `tracesSampleRate` is applied + +#### Scenario: Tracing enabled without DSN + +- **WHEN** `SENTRY_TRACING` is truthy but `SENTRY_DSN` is empty or unset +- **THEN** the tracer is a no-op and no spans are exported, since export requires a DSN + +### Requirement: Error capture is preserved when tracing is disabled + +The system SHALL continue to report exceptions thrown inside hot-path work wrappers to Sentry even when tracing is disabled, so disabling spans never reduces error visibility. + +#### Scenario: Exception in hot-path wrapper with tracing off + +- **WHEN** `SENTRY_TRACING` is disabled and code wrapped by the hot-path span helper throws +- **THEN** the exception is still captured and reported to Sentry (as an error), and re-thrown to preserve existing control flow diff --git a/openspec/changes/gate-sentry-spans/tasks.md b/openspec/changes/gate-sentry-spans/tasks.md new file mode 100644 index 0000000..d130556 --- /dev/null +++ b/openspec/changes/gate-sentry-spans/tasks.md @@ -0,0 +1,16 @@ +## 1. Tracing gate + +- [x] 1.1 Add `isTracingEnabled()` helper (reads `SENTRY_TRACING`, truthy = `"1"`/`"true"` case-insensitive) exported from the instrumentation layer +- [x] 1.2 In `src/instrumentation/otel.ts`, return the no-op tracer unless both `SENTRY_DSN` and `SENTRY_TRACING` are truthy (keep `withSpan` signature and its `Sentry.captureException` in catch) +- [x] 1.3 In `src/instrumentation/sentry.ts`, only set `tracesSampleRate` when tracing is enabled (omit it otherwise) + +## 2. Documentation + +- [x] 2.1 Add `SENTRY_TRACING` (default `false`) to `.env.example` with a note it gates spans for free-tier safety +- [x] 2.2 Update the README observability section to describe the flag and default-off behavior + +## 3. Verification + +- [x] 3.1 `npm run typecheck` passes +- [x] 3.2 `npm test` passes +- [x] 3.3 Manually confirm: with `SENTRY_TRACING` unset, no spans are emitted; error capture path unaffected diff --git a/openspec/config.yaml b/openspec/config.yaml new file mode 100644 index 0000000..392946c --- /dev/null +++ b/openspec/config.yaml @@ -0,0 +1,20 @@ +schema: spec-driven + +# Project context (optional) +# This is shown to AI when creating artifacts. +# Add your tech stack, conventions, style guides, domain knowledge, etc. +# Example: +# context: | +# Tech stack: TypeScript, React, Node.js +# We use conventional commits +# Domain: e-commerce platform + +# Per-artifact rules (optional) +# Add custom rules for specific artifacts. +# Example: +# rules: +# proposal: +# - Keep proposals under 500 words +# - Always include a "Non-goals" section +# tasks: +# - Break tasks into chunks of max 2 hours diff --git a/src/instrumentation/otel.ts b/src/instrumentation/otel.ts index 5119d36..85affac 100644 --- a/src/instrumentation/otel.ts +++ b/src/instrumentation/otel.ts @@ -2,17 +2,20 @@ import * as Sentry from "@sentry/node"; import { trace, type Span, type Tracer } from "@opentelemetry/api"; import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node"; import { SentrySpanProcessor, SentryPropagator } from "@sentry/opentelemetry"; +import { isTracingEnabled } from "./tracing.js"; let tracer: Tracer | null = null; /** - * OpenTelemetry spans export to Sentry via @sentry/opentelemetry when SENTRY_DSN is set. - * Without a DSN, spans are no-ops (safe for local dev and tests). + * OpenTelemetry spans export to Sentry via @sentry/opentelemetry only when + * SENTRY_DSN is set AND SENTRY_TRACING is enabled. Otherwise spans are no-ops + * (default; safe for 24/7 operation, local dev, and tests). Error capture in + * `withSpan` is preserved regardless. */ export function initOpenTelemetry(): Tracer { if (tracer) return tracer; - if (!process.env.SENTRY_DSN?.trim()) { + if (!process.env.SENTRY_DSN?.trim() || !isTracingEnabled()) { tracer = trace.getTracer("arbpulse"); return tracer; } diff --git a/src/instrumentation/sentry.ts b/src/instrumentation/sentry.ts index 4edf1ba..30a430b 100644 --- a/src/instrumentation/sentry.ts +++ b/src/instrumentation/sentry.ts @@ -1,4 +1,5 @@ import * as Sentry from "@sentry/node"; +import { isTracingEnabled } from "./tracing.js"; const dsn = process.env.SENTRY_DSN?.trim(); @@ -8,8 +9,13 @@ export function initSentry(): void { Sentry.init({ dsn, environment: process.env.NODE_ENV ?? "development", - tracesSampleRate: process.env.NODE_ENV === "production" ? 0.1 : 1, integrations: [Sentry.httpIntegration(), Sentry.expressIntegration()], + // Tracing/spans are gated by SENTRY_TRACING (default off) to stay within + // the Sentry free-tier span quota during 24/7 operation. Error monitoring + // is always on. + ...(isTracingEnabled() + ? { tracesSampleRate: process.env.NODE_ENV === "production" ? 0.1 : 1 } + : {}), }); process.on("unhandledRejection", (reason) => { diff --git a/src/instrumentation/tracing.ts b/src/instrumentation/tracing.ts new file mode 100644 index 0000000..e4eb5c8 --- /dev/null +++ b/src/instrumentation/tracing.ts @@ -0,0 +1,11 @@ +/** + * Single source of truth for the tracing gate. OpenTelemetry spans are only + * created and exported to Sentry when `SENTRY_TRACING` is truthy (and a + * `SENTRY_DSN` is set). Defaults to disabled so 24/7 operation stays off the + * Sentry free-tier span quota. Error monitoring is unaffected by this flag. + */ +export function isTracingEnabled(): boolean { + const raw = process.env.SENTRY_TRACING; + if (raw === undefined) return false; + return raw === "1" || raw.toLowerCase() === "true"; +}