Short answer: use trace-centered observability for a customer-support retrieval architecture, with structured events and freshness metrics attached to each trace. A log-only design is quick to ship, but it makes a bad answer hard to explain because chunking, embedding, filtering, and reranking happen in different places.

I build the ingestion path first: split support articles, attach stable metadata, embed the chunks, and write them to a vector index. The retrieval path then turns a ticket into candidate chunks for the language model. The important decision is not which dashboard looks nicest. It is whether one support answer can be reconstructed from source revision to final context.

The experiment: one ticket, two observability designs

For a small Node.js service, I compared two designs on the same flow. Design A emitted a log line after each step. Design B created one trace for the ticket and made ingestion and retrieval events children of that trace. Both recorded the query, document revision, chunk IDs, filter expression, candidate count, and model latency. Neither stored the customer's raw message by default.

The simple log approach looked fine during a quiet test. Then a document was edited while its old chunks were still searchable. Logs showed successful writes and successful reads, but joining those lines required guessing a timestamp and a request ID from three services. I don't want an on-call engineer replaying that guesswork at 2 a.m.; the useful evidence has to travel with the request. The trace showed the stale revision and the filter decision in one view, alongside the queue hand-off, embedding version, candidate count before filtering, and final chunk IDs. That single chain made it possible to say exactly where freshness was lost: publication, indexing, filtering, or ranking. It also exposed a second trap: a fast query can still be wrong when a policy article was replaced between two support shifts.

That distinction matters for freshness. A vector index can be available and fast while still serving yesterday's refund policy. I now treat source_revision, indexed_at, and retrieved_at as first-class fields, not comments in a log message.

Here is the smallest event contract I use. It is deliberately boring; boring fields survive migrations.

type RetrievalEvent = { traceId: string; stage: "chunk" | "embed" | "index" | "retrieve" | "rerank"; documentId?: string; sourceRevision?: string; chunkId?: string; candidateCount?: number; latencyMs: number; outcome: "ok" | "filtered" | "empty"; }; function emit(event: RetrievalEvent): void { process.stdout.write(JSON.stringify(event) + "\n"); } 

Before copying this choice, measure trace storage, p95 retrieval latency, empty-result rate, and the age of the revision returned for resolved tickets. Your mileage may vary: a single-process prototype may gain little from distributed traces, while a support system with asynchronous indexing usually gains a lot.

How should retrieval architecture expose customer-support knowledge-base failures?

Start with a stage map, not a vendor map. A useful trace has spans for normalization, chunking, embedding, index write, candidate retrieval, metadata filtering, reranking, and prompt assembly. Each span answers one operational question: did the data disappear, did the search miss it, or did a later step discard it?

The span attributes need guardrails. Record hash values or redacted labels for ticket text, cap attribute sizes, and keep secrets out of baggage. Keep the full source text in an access-controlled store, referenced by an ID. This is an engineering boundary, not an invitation to copy customer conversations into a telemetry backend.

For chunking, emit the source revision, splitter version, token-count bucket, and overlap policy. For retrieval, emit the index name, query embedding version, filter result, and top-k. For the final context, emit selected chunk IDs and their rank positions. A support engineer can then distinguish “the article was never indexed” from “the article ranked 12th” without replaying a production request.

OpenTelemetry gives a portable model for traces, metrics, and logs, so the application can change collectors without changing its event contract. Prometheus-style metrics are useful for trends; a trace system is useful for one ticket; a log store is useful for searching a narrow field. They overlap, but they are not interchangeable.

What signals separate stale chunks from a bad search?

Freshness needs two clocks. index_lag_seconds measures source publication to index visibility. answer_age_seconds measures publication time of the newest chunk actually used in a response. The first can be healthy while the second is poor if ranking or filters select old material.

I also track a small set of counters and distributions:

  • retrieval_empty_total, split by normalized intent and index version.
  • candidate_count, before and after metadata filters.
  • chunk_age_seconds for selected context, with p50 and p95 values.
  • rerank_drop_ratio, so an aggressive reranker is visible.
  • trace_duration_ms, split by ingestion and query paths.

An alert should point to a decision, not a raw threshold. For example, page someone when empty results rise and index lag is normal; that suggests query or filter drift. Open a data pipeline incident when lag rises across many documents. Sampling every successful trace can be expensive, so retain all empty or policy-sensitive traces and sample the ordinary successes.

The first time I added chunk_age_seconds, I found a false success: the answer citation existed, but its revision was older than the ticket's target date. That was not a model failure. It was a missing freshness assertion.

function assertFreshness( publishedAt: number, selectedAt: number, maxAgeMs: number, ): "ok" | "stale" { return selectedAt - publishedAt <= maxAgeMs ? "ok" : "stale"; } 

Why compare logs, metrics, and traces instead of picking one?

A log-first stack is easy to query and often cheap to retain, but correlation becomes application code. Metrics-first monitoring is excellent for alerting and capacity planning, yet it cannot explain why one ticket received a particular chunk. Trace-first instrumentation carries context across queues and HTTP calls, although retention and cardinality require discipline.

Three common tools illustrate the boundaries without making any one a recommendation. OpenTelemetry defines the instrumentation and context conventions. Prometheus focuses on time-series metrics and alert rules. Jaeger provides a trace exploration experience. They can be combined, replaced, or self-hosted; the durable choice is the event schema and redaction policy, not the brand on the UI.

The catch is operational weight. Traces are not suitable when telemetry cannot be kept within your support-data residency rules, or when the team has no owner for sampling and retention. Stick with structured logs plus a few metrics for a local prototype. Move to traces when asynchronous workers, multiple indexes, or audit questions make request reconstruction a weekly task.

A ship-first rollout that preserves evidence

Add a correlation ID at the edge and propagate it through the queue. Instrument ingestion before retrieval, because a missing revision is otherwise indistinguishable from a bad query. Add dashboards only after events can be inspected for one known ticket.

In CI, run a small fixture of support articles with deliberate revisions: publish v1, index it, publish v2, and assert that retrieval reports the allowed age. Include a case where metadata filtering removes every candidate. These tests exercise the failure explanations, not just the happy-path answer.

Keep the telemetry payload versioned. A field such as schema_version: 2 lets you change a splitter or embedding model without breaking historical queries. Review access to raw prompts separately from access to latency metrics, and set a deletion policy that matches the source system.

The practical decision rule is short: choose the least expensive signal set that can prove why a chunk was selected and how fresh it was. For a support knowledge base, that usually means traces for sampled requests, structured events for every stage, and metrics for fleet-wide drift. Recheck the rule when traffic, compliance, or index topology changes.

References