Exemplar-Based SLO Alerting: Linking Prometheus Burn Rate to Individual Traces in Go Services
Burn-rate SLO alerting tells you how fast your error budget is draining. It does not tell you which request caused the drain. The operational gap between those two facts is where incident diagnosis stalls: an on-call engineer sees a multi-window burn-rate alert fire, opens Grafana, confirms the ratio is ugly, then pivots to Jaeger or Tempo and starts manually correlating timestamps. That pivot is slow, lossy, and entirely avoidable.
Prometheus exemplars exist precisely to close this gap. They embed a trace ID—and optionally a span ID—directly inside a histogram observation, so the aggregated metric and the individual request that drove it are permanently linked at write time. When an SLO burn rate breaches, you can chase the exemplar on the boundary of the error budget window and land on the exact trace that tipped the threshold.
This article covers the mechanics of exemplar injection in Go, how to preserve exemplars across the Prometheus scrape pipeline, and how to build an alerting architecture that surfaces trace links at alert time rather than requiring manual correlation afterward.
Why Burn-Rate Alerts Without Exemplars Fail Operationally
A standard multiwindow, multiburn-rate alert fires when your short window (e.g., one hour) and long window (e.g., six hours) both exceed a burn-rate multiple. The Prometheus recording rule looks roughly like:
sum(rate(http_request_duration_seconds_count{job="api",code=~"5.."}[1h])) / sum(rate(http_request_duration_seconds_count{job="api"}[1h])) At alert time you know the ratio. You do not know whether the errors are concentrated in one endpoint, one pod, one upstream dependency, or distributed randomly. The ratio is an aggregate. Every piece of diagnosis you do after alert receipt is reconstruction work—querying logs, filtering by time range, guessing at exemplar candidates. The error budget has already burned. Every minute of reconstruction is compounding loss.
Exemplars let you attach the highest-latency or error-class trace ID at metric write time. The question is whether your pipeline is configured to retain and surface them.
Exemplar Mechanics in Go
Prometheus's Go client (prometheus/client_golang) exposes exemplar-aware observer types. The key type is prometheus.ExemplarObserver, satisfied by prometheus.ObserverVec returned from NewHistogramVec when you use prometheus.NewHistogramVec with an ExemplarMaxAge and appropriate bucket configuration.
The critical point: you must call ObserveWithExemplar rather than Observe, and you must supply a prometheus.Labels map that contains your trace and span IDs.
import ( "net/http" "time" "github.com/prometheus/client_golang/prometheus" "go.opentelemetry.io/otel/trace" ) var requestDuration = prometheus.NewHistogramVec( prometheus.HistogramOpts{ Name: "http_request_duration_seconds", Help: "Request latency with exemplars for SLO correlation.", Buckets: prometheus.DefBuckets, }, []string{"method", "status"}, ) func instrumentedHandler(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { start := time.Now() rw := &statusRecorder{ResponseWriter: w, status: 200} next.ServeHTTP(rw, r) elapsed := time.Since(start).Seconds() spanCtx := trace.SpanFromContext(r.Context()).SpanContext() exemplarLabels := prometheus.Labels{} if spanCtx.IsValid() { exemplarLabels["traceID"] = spanCtx.TraceID().String() exemplarLabels["spanID"] = spanCtx.SpanID().String() } obs := requestDuration.WithLabelValues(r.Method, http.StatusText(rw.status)) if eo, ok := obs.(prometheus.ExemplarObserver); ok && len(exemplarLabels) > 0 { eo.ObserveWithExemplar(elapsed, exemplarLabels) } else { obs.Observe(elapsed) } }) } Two tradeoffs are immediate.
Exemplar cardinality is capped per bucket. Each histogram bucket retains exactly one exemplar (the most recent observation that landed in that bucket). This is not a configurable ring buffer; it is a last-write-wins slot. High request rates will overwrite exemplars rapidly. If your SLO is latency-based, you want the exemplar to survive long enough for Prometheus to scrape it. Scrape intervals of 15 seconds combined with very high traffic per bucket mean the exemplar at bucket boundary (e.g., the 500ms bucket that defines your latency SLO) will be overwritten before it is scraped. Mitigation: use a dedicated histogram for SLO-critical paths with a narrower scrape interval, or downsample the write rate by sampling exemplars only for requests that exceed your SLO threshold bucket.
Sampling bias. If you inject exemplars for every request, fast requests dominate by sheer volume. You want exemplars that are diagnostically useful—requests that are slow or that returned 5xx. Filter exemplar injection:
if elapsed > sloLatencyThreshold || rw.status >= 500 { eo.ObserveWithExemplar(elapsed, exemplarLabels) } else { obs.Observe(elapsed) } This is intentional bias toward diagnostic signal. You are not sampling for accurate aggregate statistics (that is what the base counter does); you are sampling for navigable traces.
Pipeline Preservation: Scrape, TSDB, and Remote Write
Exemplars are useless if the pipeline discards them before they reach your query layer.
Prometheus scrape config: Exemplars are scraped by default when the target exposes them via the OpenMetrics text format (not the classic Prometheus format). Verify your Go service is serving application/openmetrics-text when the scraper requests it. promhttp.HandlerFor with promhttp.HandlerOpts{EnableOpenMetrics: true} is required; otherwise exemplars are silently dropped at the scrape boundary.
TSDB retention: Prometheus's TSDB stores exemplars in a separate circular buffer (--storage.tsdb.exemplar-storage.max-exemplars). The default is 100,000 exemplars across all series. For a high-traffic service with SLO-critical histograms, this fills quickly. Calculate your required retention: scrape interval × series count × buckets per series × requests per second. If your burn-rate alert windows span six hours, you need exemplar retention that covers at least the long window. Size accordingly or accept that older exemplar links will be evicted before alert diagnosis.
Remote write to Grafana Mimir or Cortex: Exemplars propagate over remote write only if the remote write target supports them (remote_write.send_exemplars: true in Prometheus 2.31+). Mimir and Cortex both support exemplar ingestion natively. Thanos does not propagate exemplars through its store API as of mid-2026 without a sidecar configuration change. Verify your long-term storage path before depending on exemplar availability during postmortems.
Wiring the Alert to a Trace Link
The goal is that when a burn-rate alert fires, the alert message contains a direct link to a trace. The mechanism is a Prometheus alerting rule that queries the exemplar for the SLO-critical bucket at the time the alert fires.
Prometheus does not natively embed exemplar values in alert annotations via PromQL—exemplar is not a PromQL function. The practical approach is two-pronged.
First, configure your alertmanager template to append a pre-built Tempo/Jaeger query URL using the time range of the alert window:
annotations: summary: "Error budget burning at {{ $value }}x rate" trace_search: >- https://tempo.internal/search?start={{ $startsAt | unixEpoch }}&end={{ $endsAt | unixEpoch }}&service=api&minDuration=500ms This is a search link, not a direct trace link, because PromQL cannot dereference exemplar label values at alert evaluation time.
Second, for direct trace links, run a small sidecar service—call it the exemplar linker—that subscribes to alertmanager webhook notifications, queries the Prometheus HTTP API's /api/v1/query_exemplars endpoint for the relevant metric and time range, extracts the trace ID from the exemplar closest to the alert start time, and enriches the alert payload before routing it to PagerDuty or Slack:
func enrichAlert(ctx context.Context, alert Alert, promClient api.Client) (string, error) { q := `http_request_duration_seconds_bucket{job="api",le="0.5",status="Internal Server Error"}` start := alert.StartsAt.Add(-5 * time.Minute) end := alert.StartsAt.Add(time.Minute) result, _, err := promClient.QueryRange(ctx, q, v1.Range{ Start: start, End: end, Step: 30 * time.Second, }) // query_exemplars endpoint exemplars, _, err := promClient.QueryExemplars(ctx, q, start, end) if err != nil || len(exemplars) == 0 { return "", err } // find exemplar closest to alert start best := closestExemplar(exemplars, alert.StartsAt) traceID := best.Labels["traceID"] return fmt.Sprintf("https://tempo.internal/trace/%s", traceID), nil } This sidecar runs as a stateless AWS Lambda or ECS task. It adds one network round-trip to alert delivery but eliminates the manual correlation step entirely.
Operational Tradeoffs and Decision Framework
Use exemplar injection when: your SLO is latency or error-rate based, your trace backend is queryable by trace ID (Tempo, Jaeger, Zipkin), and your Prometheus pipeline runs OpenMetrics scraping end-to-end. The implementation cost is low; the diagnostic value per alert is high.
Do not depend on exemplars alone when: request rates per bucket are so high that last-write-wins semantics guarantee the exemplar at alert time is unrelated to the error causing the burn. In this regime, exemplar sampling at the application layer (injecting only on threshold-crossing requests) partially compensates, but you should also maintain structured logs with trace IDs and use log-based correlation as a fallback.
Size exemplar storage to your longest alert window. If your six-hour burn-rate alert fires and exemplar storage covers only two hours, you cannot diagnose the beginning of the event. Budget --storage.tsdb.exemplar-storage.max-exemplars based on peak series × buckets × scrape rate × window duration.
Prefer the exemplar linker sidecar over templated search links when your on-call team's mean time to diagnosis (MTTD) is a tracked SLI. A direct trace link reduces MTTD measurably. A search link requires an additional manual step that compounds under alert fatigue.
Validate exemplar propagation in staging. Prometheus's OpenMetrics negotiation, TSDB exemplar retention, and remote write exemplar forwarding each have independent on/off switches. It is routine to ship a correctly instrumented Go service and discover in production that exemplars are silently discarded at the scrape or remote write boundary. Write a smoke test that scrapes your staging service, queries /api/v1/query_exemplars, and asserts at least one exemplar exists for your SLO histogram after synthetic load.
Burn-rate SLO alerting without trace linkage is a ratio with no context. Exemplars make the ratio navigable. The implementation is shallow; the pipeline configuration is where production deployments fail. Get the pipeline right, and every SLO breach hands the on-call engineer the request that broke it.
