feat(observability): gate Sentry tracing spans behind SENTRY_TRACING

Spans are now created and exported to Sentry only when SENTRY_TRACING is enabled (default off), keeping 24/7 operation within the free-tier span quota. Error monitoring stays always-on. Includes OpenSpec change gate-sentry-spans (proposal, design, specs, tasks).

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
Mauricio Barragan
2026-07-16 12:54:15 -06:00
parent f6e8381c20
commit a5c83ee026
11 changed files with 169 additions and 5 deletions
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-07-16
@@ -0,0 +1,34 @@
## Context
`src/instrumentation/otel.ts` builds a tracer that, when `SENTRY_DSN` is set, registers a `NodeTracerProvider` with `SentrySpanProcessor`/`SentryPropagator`, so every `withSpan(...)` call on the hot path creates and exports a span to Sentry. `src/instrumentation/sentry.ts` sets `tracesSampleRate` unconditionally. Running 24/7 on a VPS multiplies span volume against Sentry's free-tier quota (5M spans/month), while error events stay small. We want tracing to remain a debugging option but be off by default.
## Goals / Non-Goals
**Goals:**
- Add a single boolean env var (`SENTRY_TRACING`, default `false`) that turns span creation/export on or off.
- Default behavior emits zero spans and zero hot-path span allocation.
- Preserve error monitoring exactly as-is regardless of the flag.
- No dependency changes; no API/wire changes.
**Non-Goals:**
- Removing OpenTelemetry or the `withSpan` abstraction.
- Making `tracesSampleRate` itself env-configurable (kept as current 0.1 prod / 1 dev when enabled).
- Changing which operations are wrapped.
## Decisions
- **Boolean gate over sample-rate gate.** A boolean `SENTRY_TRACING` that makes the tracer a no-op avoids hot-path span allocation entirely when off (aligns with the "no unnecessary alloc on the hot path" convention). A `tracesSampleRate=0` approach would still register the provider and create spans that are dropped later — more overhead. Alternative considered and rejected.
- **Gate lives in `otel.ts` init.** `initOpenTelemetry()` returns the no-op `trace.getTracer("arbpulse")` unless both `SENTRY_DSN` and `SENTRY_TRACING` are truthy. This keeps `withSpan`'s signature and all call sites unchanged; its `catch` still calls `Sentry.captureException`, so errors are captured even with a no-op span.
- **Conditional `tracesSampleRate` in `sentry.ts`.** Only set `tracesSampleRate` when tracing is enabled; omit it otherwise so Sentry performance is fully off. A shared helper `isTracingEnabled()` reads the flag so both files agree.
- **Parsing.** Reuse the existing truthy convention (`"1"` or `"true"`, case-insensitive) used by `config.ts`'s `bool()` for consistency; implement locally in the instrumentation module since it initializes before `config` import.
## Risks / Trade-offs
- [Someone enables tracing in prod and hits the quota] → Documented in `.env.example`/README that it is off by default and why; enabling is an explicit opt-in.
- [Flag read in two files could drift] → Centralize in one `isTracingEnabled()` helper exported from the instrumentation layer.
- [No-op tracer still wraps hot path in a promise] → Accepted: `withSpan` remains async as today; the change only removes span export, matching prior behavior when `SENTRY_DSN` was unset.
## Migration Plan
- Deploy with `SENTRY_TRACING` unset → tracing off (safe default), errors still reported.
- To debug, set `SENTRY_TRACING=true` (with `SENTRY_DSN`) temporarily; unset to roll back. No data migration.
@@ -0,0 +1,25 @@
## Why
Arb Pulse is moving to a 24/7 VPS (Hetzner), where the engine runs continuously and emits OpenTelemetry spans on the hot path (`ws.message``orderbook.process``arbitrage.evaluate``sse.broadcast`). At that volume the app can approach or exceed Sentry's free-tier span quota (5M spans/month), while error monitoring stays well within limits. We want to keep tracing available for debugging but off by default so continuous operation is free-tier safe.
## What Changes
- Introduce an environment variable (`SENTRY_TRACING`, boolean, default `false`) that gates OpenTelemetry span creation and export to Sentry.
- When `SENTRY_TRACING` is disabled (default), the tracer is a no-op: no spans are created or exported, and no hot-path span allocation occurs. Error capture (`Sentry.captureException`, `unhandledRejection`, `uncaughtException`) remains fully active.
- When `SENTRY_TRACING` is enabled **and** `SENTRY_DSN` is set, spans are created and exported to Sentry as today, and `tracesSampleRate` is applied.
- Document the new variable in `.env.example` and `README.md`.
## Capabilities
### New Capabilities
- `observability`: error monitoring and optional distributed tracing behavior — how Sentry error capture is always on, and how OpenTelemetry span emission is conditioned on an environment flag.
### Modified Capabilities
<!-- None: no existing specs under openspec/specs/. -->
## Impact
- Code: `src/instrumentation/otel.ts` (gate span provider/export), `src/instrumentation/sentry.ts` (conditional `tracesSampleRate`), possibly `src/instrumentation/index.ts`.
- Config/docs: `.env.example`, `README.md` (observability section + env var table).
- Dependencies: none added or removed (OTel packages stay for the enabled path).
- Behavior: default runtime emits zero spans; error monitoring unchanged. No API or wire-format changes.
@@ -0,0 +1,43 @@
## ADDED Requirements
### Requirement: Error monitoring is always active
The system SHALL report runtime errors to Sentry whenever `SENTRY_DSN` is configured, independent of any tracing configuration. This includes REST, WebSocket feed, and SSE errors captured via `Sentry.captureException`, plus process-level `unhandledRejection` and `uncaughtException` handlers.
#### Scenario: Error captured with DSN set
- **WHEN** `SENTRY_DSN` is set and a handled error occurs on the WebSocket, REST, or SSE path
- **THEN** the error is sent to Sentry as an error event
#### Scenario: No DSN configured
- **WHEN** `SENTRY_DSN` is empty or unset
- **THEN** Sentry is not initialized and no events are sent, and the application continues running normally
### Requirement: Tracing spans are gated by an environment flag
The system SHALL only create and export OpenTelemetry spans when the `SENTRY_TRACING` environment variable is enabled AND `SENTRY_DSN` is set. The flag SHALL default to disabled so that continuous 24/7 operation emits zero spans by default.
#### Scenario: Tracing disabled by default
- **WHEN** `SENTRY_TRACING` is unset or falsy (default)
- **THEN** the tracer is a no-op, no spans are created on the hot path, and no spans are exported to Sentry
#### Scenario: Tracing enabled with DSN
- **WHEN** `SENTRY_TRACING` is truthy and `SENTRY_DSN` is set
- **THEN** hot-path spans (`ws.message`, `orderbook.process`, `arbitrage.evaluate`, `sse.broadcast`) are created and exported to Sentry, and `tracesSampleRate` is applied
#### Scenario: Tracing enabled without DSN
- **WHEN** `SENTRY_TRACING` is truthy but `SENTRY_DSN` is empty or unset
- **THEN** the tracer is a no-op and no spans are exported, since export requires a DSN
### Requirement: Error capture is preserved when tracing is disabled
The system SHALL continue to report exceptions thrown inside hot-path work wrappers to Sentry even when tracing is disabled, so disabling spans never reduces error visibility.
#### Scenario: Exception in hot-path wrapper with tracing off
- **WHEN** `SENTRY_TRACING` is disabled and code wrapped by the hot-path span helper throws
- **THEN** the exception is still captured and reported to Sentry (as an error), and re-thrown to preserve existing control flow
@@ -0,0 +1,16 @@
## 1. Tracing gate
- [x] 1.1 Add `isTracingEnabled()` helper (reads `SENTRY_TRACING`, truthy = `"1"`/`"true"` case-insensitive) exported from the instrumentation layer
- [x] 1.2 In `src/instrumentation/otel.ts`, return the no-op tracer unless both `SENTRY_DSN` and `SENTRY_TRACING` are truthy (keep `withSpan` signature and its `Sentry.captureException` in catch)
- [x] 1.3 In `src/instrumentation/sentry.ts`, only set `tracesSampleRate` when tracing is enabled (omit it otherwise)
## 2. Documentation
- [x] 2.1 Add `SENTRY_TRACING` (default `false`) to `.env.example` with a note it gates spans for free-tier safety
- [x] 2.2 Update the README observability section to describe the flag and default-off behavior
## 3. Verification
- [x] 3.1 `npm run typecheck` passes
- [x] 3.2 `npm test` passes
- [x] 3.3 Manually confirm: with `SENTRY_TRACING` unset, no spans are emitted; error capture path unaffected