Tail Sampling with OpenTelemetry, Measured: It Keeps What Matters Until You Scale It or Shrink It
Logs, Metrics and Traces Are Not Three Pillars. They're One Request. ended on an uncomfortable number. A 10% head sampler decides whether to keep a trace when the request arrives, before anyone knows it will be slow. So of 20 slow checkouts, it kept about 2, and the histogram that told you about the slow ones pointed at traces that no longer existed. I promised the fix in that post, and again in the structured logging one: decide at the end of the trace instead. That is tail sampling, and this is the post.
Tail sampling does what it promises. On my run it kept every slow trace and every failed one, while dropping 90% of the rest. It also has two failure modes that a single-Collector demo never shows, and both lose exactly the traces you added it for: a buffer that is too small, and a second Collector.
Everything here runs locally: three Node services, the OpenTelemetry Collector (contrib 0.164.0) and Jaeger 2.22, in Docker.
The system, and how to know what was kept
Aurora Coffee Co.'s checkout calls pricing, then inventory. Two things go wrong on purpose, at known rates, so every result can be checked against the truth:
- 1 order in 50 is slow. Seasonal SKUs make pricing wait 600 ms on its promotion rules.
- 1 order in 100 fails. One inventory warehouse shard answers 503, and checkout answers 502.
The services are plain node:http. Node 24 and later run these TypeScript files directly, with no build step:
{ "name": "aurora-tail-sampling", "private": true, "type": "module", "engines": { "node": ">=24" }, "dependencies": { "@opentelemetry/api": "1.10.0", "@opentelemetry/auto-instrumentations-node": "0.82.0", "@opentelemetry/exporter-trace-otlp-http": "0.300.0", "@opentelemetry/sdk-node": "0.300.0", "import-in-the-middle": "3.5.3" } } // pricing.ts — quotes a price. Seasonal SKUs wait on a slow promotion-rules lookup. import { createServer } from 'node:http' import { setTimeout as sleep } from 'node:timers/promises' const port = Number(process.env.PORT ?? 3001) createServer(async (req, res) => { const sku = new URL(req.url ?? '/', 'http://pricing').searchParams.get('sku') ?? '' // Stands in for the real cost: 2,400 promotion rules against 12 for a house blend. if (sku.startsWith('seasonal-')) await sleep(600) res.writeHead(200, { 'content-type': 'application/json' }) res.end(JSON.stringify({ sku, cents: sku.startsWith('seasonal-') ? 2100 : 1600 })) }).listen(port) // inventory.ts — reserves stock. One warehouse shard is unavailable for 1 order in 100. import { createServer } from 'node:http' const port = Number(process.env.PORT ?? 3002) createServer((req, res) => { const order = Number(new URL(req.url ?? '/', 'http://inventory').searchParams.get('order')) if (order % 100 === 37) { res.writeHead(503, { 'content-type': 'application/json' }) res.end(JSON.stringify({ error: 'warehouse shard unavailable' })) return } res.writeHead(200, { 'content-type': 'application/json' }) res.end(JSON.stringify({ order, reserved: true })) }).listen(port) // checkout.ts — the front door: prices the order, then reserves stock. import { createServer } from 'node:http' import { trace } from '@opentelemetry/api' const port = Number(process.env.PORT ?? 3000) const pricingUrl = process.env.PRICING_URL ?? 'http://localhost:3001' const inventoryUrl = process.env.INVENTORY_URL ?? 'http://localhost:3002' async function call(url: string): Promise<unknown> { const response = await fetch(url, { signal: AbortSignal.timeout(5_000) }) if (!response.ok) throw new Error(`${url} answered ${response.status}`) return response.json() } createServer(async (req, res) => { const order = Number(new URL(req.url ?? '/', 'http://checkout').searchParams.get('order')) // One order in 50 is a seasonal SKU, the slow path in pricing. const sku = order % 50 === 0 ? `seasonal-${order}` : `house-${order % 7}` // Returned so the load generator can look the trace up in Jaeger afterwards. res.setHeader('x-trace-id', trace.getActiveSpan()?.spanContext().traceId ?? '') try { const price = await call(`${pricingUrl}/quote?sku=${sku}`) await call(`${inventoryUrl}/reserve?order=${order}`) res.writeHead(200, { 'content-type': 'application/json' }) res.end(JSON.stringify({ order, price })) } catch (error) { res.writeHead(502, { 'content-type': 'application/json' }) res.end(JSON.stringify({ order, error: error instanceof Error ? error.message : 'checkout failed' })) } }).listen(port) The tracing setup is two small files, loaded before the service with two --import flags. The first installs the loader hook that lets auto-instrumentation patch ES modules:
// register.ts — loaded first: `node --import ./register.ts --import ./telemetry.ts service.ts`. import { register } from 'node:module' import { register as registerSyncHooks, supportsSyncHooks } from 'import-in-the-middle/register-hooks.mjs' // Node 24.11.1+ and 26 run the hook in-thread with module.registerHooks(). module.register(), // the older off-thread loader, is deprecated in Node 26 and kept only as the fallback. if (supportsSyncHooks()) registerSyncHooks() else register('@opentelemetry/instrumentation/hook.mjs', import.meta.url) The second starts the SDK:
// telemetry.ts — loaded second, after register.ts, and before any service code. import { NodeSDK } from '@opentelemetry/sdk-node' import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node' import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http' const sdk = new NodeSDK({ serviceName: process.env.OTEL_SERVICE_NAME ?? 'unnamed-service', // Keep host names, user names and PIDs out of every span. autoDetectResources: false, traceExporter: new OTLPTraceExporter({ url: `${process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? 'http://localhost:4318'}/v1/traces`, }), instrumentations: [getNodeAutoInstrumentations({ '@opentelemetry/instrumentation-fs': { enabled: false } })], }) sdk.start() for (const signal of ['SIGINT', 'SIGTERM'] as const) { process.once(signal, () => { void sdk.shutdown().finally(() => process.exit(0)) }) } The hook is the part that changed since September, and it is now a file of its own for a reason. Without a hook, auto-instrumentation never sees node:http under ESM and every service starts its own trace. The September post used module.register(), which Node 26 deprecates with a warning on every start. import-in-the-middle, the library behind OpenTelemetry's hook, ships an in-thread version built on module.registerHooks(), and supportsSyncHooks() says whether the running Node can use it. It has to run in its own --import: static import statements are evaluated before the code of the file that contains them, so a hook registered at the top of telemetry.ts would run after that file's imports had already loaded.
The ground truth
To score a sampler you need to know which requests were slow or failed, independently of the traces. The load generator records that from the client side, along with each request's trace ID, which checkout returns in a header:
// load.ts — sends N checkouts and records what really happened to each one. // Usage: node load.ts <requests> <concurrency> <out.json> import { writeFileSync } from 'node:fs' type Outcome = { order: number; traceId: string; status: number; ms: number } const [requests = '1000', concurrency = '10', out = 'outcomes.json'] = process.argv.slice(2) const total = Number(requests) const outcomes: Outcome[] = [] let next = 1 async function worker(): Promise<void> { while (next <= total) { const order = next++ const started = performance.now() const response = await fetch(`http://localhost:3000/checkout?order=${order}`) await response.arrayBuffer() outcomes.push({ order, traceId: response.headers.get('x-trace-id') ?? '', status: response.status, ms: Math.round(performance.now() - started) }) } } const started = performance.now() await Promise.all(Array.from({ length: Number(concurrency) }, worker)) writeFileSync(out, JSON.stringify(outcomes)) const slow = outcomes.filter((o) => o.ms >= 500).length const failed = outcomes.filter((o) => o.status >= 500).length console.log(`${outcomes.length} checkouts in ${((performance.now() - started) / 1000).toFixed(1)} s: ${slow} over 500 ms, ${failed} failed`) Then a second script asks Jaeger, trace ID by trace ID, whether it has each trace, and whether the trace is whole. A checkout in this system is exactly five spans: checkout's server span, its two outgoing calls, and the pricing and inventory server spans. That second question matters later.
// measure.ts — asks Jaeger which of the recorded traces it actually has, and whether they are whole. // Usage: node measure.ts <outcomes.json> import { readFileSync } from 'node:fs' type Outcome = { order: number; traceId: string; status: number; ms: number } type JaegerTrace = { spans: unknown[] } const outcomes: Outcome[] = JSON.parse(readFileSync(process.argv[2] ?? 'outcomes.json', 'utf8')) const jaeger = process.env.JAEGER_URL ?? 'http://localhost:16686' // Checkout's server span, its two fetch calls, and the pricing and inventory server spans. const spansPerCheckout = 5 async function lookup(traceId: string): Promise<JaegerTrace | null> { const response = await fetch(`${jaeger}/api/traces/${traceId}`) if (response.status === 404) return null const body = (await response.json()) as { data?: JaegerTrace[] } return body.data?.[0] ?? null } const rows = new Map<string, { requests: number; kept: number; whole: number }>() for (const outcome of outcomes) { const kind = outcome.status >= 500 ? 'failed' : outcome.ms >= 500 ? 'slow' : 'normal' const row = rows.get(kind) ?? { requests: 0, kept: 0, whole: 0 } row.requests++ const found = await lookup(outcome.traceId) if (found) { row.kept++ if (found.spans.length === spansPerCheckout) row.whole++ } rows.set(kind, row) } console.log('kind requests in Jaeger whole') for (const kind of ['slow', 'failed', 'normal']) { const row = rows.get(kind) ?? { requests: 0, kept: 0, whole: 0 } console.log(`${kind.padEnd(9)} ${String(row.requests).padStart(8)} ${String(row.kept).padStart(9)} ${String(row.whole).padStart(5)}`) } The stack
One Compose file covers every scenario in this post. Each one is a profile, so you bring up exactly the Collectors it needs:
# One Collector image, five roles. `docker compose --profile <name> up -d` picks the scenario. x-collector: &collector image: otel/opentelemetry-collector-contrib:0.164.0 command: ["--config=/etc/otelcol/config.yaml"] # memory_limiter counts Go heap; this limit counts the whole process, which ran at about # 1.35x the heap in these runs. 2 GiB leaves room above limit_mib for that difference. mem_limit: 2048m services: jaeger: image: jaegertracing/jaeger:2.22.0 ports: - "16686:16686" # head: the SDK decides, this Collector only forwards. forward: <<: *collector profiles: [head] volumes: ["./collector-forward.yaml:/etc/otelcol/config.yaml:ro"] ports: ["4318:4318"] # tail: one Collector decides after seeing the whole trace. tail: <<: *collector profiles: [tail] volumes: ["${TAIL_CONFIG:-./collector-tail.yaml}:/etc/otelcol/config.yaml:ro"] ports: ["4318:4318", "8888:8888"] # split: two tail samplers, each fed directly by the services that point at it. tail-a: <<: *collector profilesSource：https://dev.to/jgomezdev/tail-sampling-with-opentelemetry-measured-it-keeps-what-matters-until-you-scale-it-or-shrink-it-46m5Related posts
SMS OTP API Error Handling Through Recipient Suppression and Bounded Evidence
What OAuth on a live remote MCP server actually looks like
Git Status Slow in a Large Repo? Two Config Lines Cut It From 17.6s to 0.83s
Structuring a 312-City Local Directory for SEO Without Creating Thin Pages
Selenium Alternatives in 2026: 7 Tools Compared by What They Fix
