Observability for AI Agents: Tracing, Evaluation and Cost Control in Production

How to understand, monitor, debug, and optimize agentic AI systems using OpenTelemetry and modern AI observability practices.

1. Why Is Observability Different for AI Agents?

Traditional backend observability is built around relatively predictable execution paths.

A request enters an application, passes through several services, interacts with a database, and eventually returns a response. We can use logs, metrics, and distributed traces to understand what happened and where a problem occurred.

AI agents introduce a different level of complexity.

An agent does not necessarily follow a predefined execution path. It may call an LLM multiple times, select tools dynamically, retry failed operations, delegate tasks to other agents, retrieve external context, or wait for additional user input.

Consider a relatively simple task:

A user asks an AI assistant to analyze a merge request and identify potential problems.

Behind the scenes, the application might:

1. Receive the user's request.
2. Invoke a coordinator agent.
3. Call an LLM to determine the next action.
4. Retrieve the merge request from GitLab.
5. Read the changed files.
6. Delegate security analysis to another agent.
7. Execute additional tool calls.
8. Evaluate the collected results.
9. Generate a final report.

The user sees one response. The application has actually executed a multi-step workflow involving several components.

Now imagine the response takes 45 seconds instead of the expected 8 seconds.

Where should we investigate?

• Did the model take too long to respond?
• Did a tool call time out?
• Was the same operation executed repeatedly?
• Did the agent generate unnecessary steps?
• Did a subagent introduce additional latency?
• Was too much context sent to the model?
• Did a retry increase the overall execution time and cost?

Traditional request-level logging may tell us that the endpoint took 45 seconds. It does not necessarily explain why.

AI observability is about understanding the entire execution path, not just measuring the final API response.

This is where OpenTelemetry becomes particularly useful.

2. What Is OpenTelemetry?

** is an open-source observability framework for collecting, processing, and exporting telemetry data.

It provides a vendor-neutral instrumentation model that allows applications to generate telemetry and send it to different observability platforms.

Instead of implementing a separate monitoring integration for every vendor, we can instrument an application using OpenTelemetry and choose where the telemetry should be exported.

OpenTelemetry is built around three primary signals:

These signals complement each other.

A metric can tell us that the average execution time has increased. A trace can identify which operation introduced the delay. A log can provide the error details associated with that operation.

OpenTelemetry also provides context propagation, APIs, SDKs, semantic conventions, and the OpenTelemetry Collector.

An important implementation detail: as of November 2026, the OpenTelemetry JavaScript documentation lists traces and metrics as Stable, while logs remain in Development. The maturity of a specific GenAI convention should also be evaluated separately from the maturity of the underlying OTel signal.

References: OpenTelemetry Documentation · OpenTelemetry JavaScript

3. Understanding Distributed Tracing

Before discussing AI-specific instrumentation, we need to understand traces and spans.

A trace represents the complete execution path of an operation.

A span represents one individual unit of work within that execution.

For example:

Trace: POST /api/agent | +-- HTTP Request | +-- Coordinator Agent | +-- LLM Generation | +-- Tool: GitLab - Get Merge Request | +-- Tool: GitLab - Get Changed Files | +-- Security Agent | | | +-- LLM Generation | | | +-- Tool: Security Analysis | +-- Testing Agent | | | +-- LLM Generation | | | +-- Tool: Test Coverage | +-- Final Response Generation 

Each span has information such as:

• Trace ID
• Span ID
• Parent span ID
• Start and end timestamps
• Duration
• Attributes
• Events
• Status and exception information

This creates a hierarchical representation of the complete execution.

The key benefit is that we can move from a high-level workflow to an individual model call or tool execution without losing the relationship between them.

For AI agents, this distinction is important because one user request can generate many independent operations.

Why is context propagation important?

In distributed systems, a trace often crosses multiple services.

Imagine a Next.js application calling a .NET backend, which then delegates a task to a Python agent and receives a result from another service.

Without context propagation, every service may generate its own independent trace.

We would see several unrelated operations instead of one complete workflow.

OpenTelemetry uses W3C Trace Context as its standard propagation format. The traceparent header carries trace and parent-span information between instrumented services.

The same principle applies to agent-based architectures.

When an agent delegates a task to another agent through A2A, we should preserve the correlation between the originating request and the remote execution.

Automatic HTTP instrumentation can handle propagation when the HTTP instrumentation and context manager are configured correctly. Custom transports and asynchronous message queues may require explicit context injection and extraction.

The important point is that protocol-level interoperability and observability context propagation are separate concerns. Using A2A does not, by itself, guarantee that all distributed spans will appear in one trace.

References: OpenTelemetry Traces · Context Propagation · W3C Trace Context

4. OpenTelemetry and GenAI Semantic Conventions

One of the challenges in AI observability is the lack of consistent naming and structure across different AI frameworks and providers.

For example, one framework might represent an LLM request as a generic operation, while another might expose model-specific attributes.

This makes it difficult to compare different implementations using the same monitoring platform.

OpenTelemetry addresses this through Semantic Conventions.

Semantic conventions define common attribute names, operation types, and instrumentation expectations so that telemetry from different implementations can be interpreted consistently.

The GenAI semantic conventions extend this approach to generative AI applications.

Some important attributes include:

For example, an LLM invocation could be represented by:

{ "gen_ai.operation.name": "chat", "gen_ai.provider.name": "openai", "gen_ai.request.model": "example-model", "gen_ai.usage.input_tokens": 1200, "gen_ai.usage.output_tokens": 350 } 

This is a simplified span attribute example.

The benefit is that observability platforms do not need to understand a different custom schema for every application.

Agent-specific conventions

The GenAI semantic conventions also define operations for agents and workflows.

Some examples include:

• invoke_agent
• invoke_workflow
• execute_tool
• plan
• retrieval

This is particularly valuable for multi-agent architectures.

We can distinguish an individual model request from an entire agent invocation or a workflow coordinating multiple agents.

For example:

invoke_workflow merge-request-review | +-- invoke_agent coordinator | | | +-- chat model | +-- execute_tool fetch_merge_request | +-- invoke_agent security-agent | | | +-- chat model | +-- execute_tool analyze_vulnerabilities | +-- invoke_agent testing-agent | | | +-- chat model | +-- execute_tool analyze_test_coverage | +-- Final Aggregation 

This makes the execution model much easier to understand.

An important maturity consideration: As of November 2, 2026, the OpenTelemetry GenAI agent spans and GenAI metrics documents are still marked Development. These conventions are valuable design references, but teams should pin their instrumentation versions and avoid treating every proposed attribute or metric as a stable, universally implemented contract.

The current convention documents include:

• Semantic Conventions for GenAI Spans
• Semantic Conventions for Agent Spans
• Semantic Conventions for GenAI Metrics

5. Integrating OpenTelemetry with Vercel AI SDK

Since many TypeScript applications use the Vercel AI SDK to implement model interactions and agent workflows, it is worth looking at its native telemetry integration.

The AI SDK already supports OpenTelemetry-based instrumentation.

It can generate spans for:

• Text generation
• Streaming generation
• Individual model calls
• Agent execution
• Tool execution
• Embedding operations
• Reranking

This means we do not need to manually instrument every model call and tool execution just to obtain basic execution visibility.

Step 1: Install the packages

For an AI SDK v7 application:

pnpm add @ai-sdk/otel @vercel/otel 

The OpenTelemetry integration is provided through @ai-sdk/otel.

This is an important version-specific detail.

In AI SDK v7, OpenTelemetry integration moved out of the core ai package. Older examples using the v6 experimental_telemetry configuration should not be copied directly into a v7 application.

Step 2: Register telemetry in Next.js

Create an instrumentation.ts file in the project root.

import { registerOTel } from '@vercel/otel'; import { registerTelemetry } from 'ai'; import { OpenTelemetry } from '@ai-sdk/otel'; export function register() { registerOTel({ serviceName: 'ai-agent-platform', }); registerTelemetry( new OpenTelemetry({ recordInputs: false, recordOutputs: false, }), ); } 

There are two responsibilities here:

• registerOTel initializes the OpenTelemetry infrastructure for the Next.js application.
• registerTelemetry registers the AI SDK-specific integration that creates GenAI spans.

I have disabled input and output capture in this example intentionally.

AI SDK telemetry can record prompts and generated outputs by default. In an enterprise environment, capturing this data requires an explicit privacy and security decision.

We will return to this in the security section.

For a regular Node.js application without Next.js, register the OpenTelemetry integration once at application startup.

Step 3: Instrument AI SDK calls

Once telemetry is registered, supported AI SDK operations automatically emit telemetry through the registered integration.

We can also provide a functionId to identify the business operation or agent.

import { generateText } from 'ai'; const result = await generateText({ model: myModel, instructions: ` You are a financial analysis assistant. Analyze the supplied information carefully. `, prompt: 'Analyze the latest portfolio performance.', telemetry: { functionId: 'portfolio-analysis-agent', }, }); 

The functionId helps distinguish operations in traces.

For example, rather than seeing only generic model invocation spans, we can identify a specific business operation such as:

invoke_agent portfolio-analysis-agent 

Step 4: Understand the generated spans

The AI SDK v7 OpenTelemetry integration follows the GenAI semantic conventions.

For generateText and streamText, the integration records three main types of spans:

1. Agent invocation span

The root span represents the complete generation operation, including its execution steps and tool calls.

Example:

invoke_agent portfolio-analysis-agent 

2. Step and model-call spans

Each model interaction is represented by step and provider-call spans.

These allow us to understand how many model calls occurred during an operation and how long each one took.

3. Tool execution spans

Each client-side tool execution receives its own span.

For example:

invoke_agent portfolio-analysis-agent | +-- step 0 | | | +-- chat example-model | +-- execute_tool get_portfolio | +-- step 1 | | | +-- chat example-model | +-- execute_tool get_market_data | +-- step 2 | +-- chat example-model 

The resulting trace provides a much more complete representation of an agent workflow than a single model request log.

The integration also records usage information such as input tokens, output tokens, cache usage, model identifiers, and operation duration when these values are available.

Step 5: Add additional application context

Sometimes the standard GenAI attributes are not enough.

For example, we may want to know which business workflow initiated an operation or which application component owns it.

The AI SDK provides enrichSpan for adding custom attributes:

import { registerTelemetry } from 'ai'; import { OpenTelemetry } from '@ai-sdk/otel'; registerTelemetry( new OpenTelemetry({ recordInputs: false, recordOutputs: false, enrichSpan: ({ spanType, operationId }) => ({ 'app.component': 'portfolio-assistant', 'app.operation_id': operationId, 'app.span_type': spanType, }), }), ); 

The exact context available to enrichSpan depends on the span and the SDK's integration API.

The objective is to enrich the trace with useful application-level context without copying sensitive business data into telemetry.

References: AI SDK Telemetry · Next.js OpenTelemetry · AI SDK v7 Migration Guide

6. Instrumenting Your Own Agent Runtime

Built-in instrumentation is useful, but it does not automatically understand every application-level decision.

For example, an AI SDK trace may show model calls and tool executions, but it may not tell us why a coordinator selected a particular specialist agent or how long a business workflow spent waiting for an approval.

These are application-level operations.

We should instrument them explicitly.

OpenTelemetry provides a tracing API for creating custom spans.

Consider a custom agent orchestration function:

import { trace, SpanStatusCode, } from '@opentelemetry/api'; const tracer = trace.getTracer('agent-orchestrator'); export async function runReviewWorkflow() { return tracer.startActiveSpan( 'workflow.merge-request-review', async (span) => { span.setAttribute( 'workflow.name', 'merge-request-review', ); span.setAttribute( 'workflow.version', '1.0.0', ); try { const result = await executeReview(); span.setStatus({ code: SpanStatusCode.OK, }); return result; } catch (cause) { const error = cause instanceof Error ? cause : new Error(String(cause)); span.recordException(error); span.setStatus({ code: SpanStatusCode.ERROR, }); throw error; } finally { span.end(); } }, ); } 

Any supported AI SDK operations executed inside this active span can become child operations in the same trace, provided the OpenTelemetry context is correctly propagated.

We now have two levels of observability:

• Framework-level instrumentation describes model and tool execution.
• Application-level instrumentation describes business workflows and orchestration decisions.

This distinction is important.

We should not expect a model instrumentation library to understand the complete business meaning of an application request.

Instrumenting remote agent calls

For A2A-based systems, distributed tracing becomes even more important.

Imagine a coordinator running in one service and a security agent running in another service.

The coordinator should create a child operation for the delegation. The remote service should continue the trace when the execution context is propagated.

Conceptually:

Trace: merge-request-review | +-- Coordinator Agent | | | +-- Model Call | | | +-- A2A: Delegate to Security Agent | | | +-- Security Agent | | | +-- Model Call | | | +-- Tool Execution | +-- Final Aggregation 

The HTTP request must carry the appropriate trace context, typically through W3C traceparent and tracestate headers.

For custom transports or asynchronous queues, explicitly inject and extract the OpenTelemetry context.

This is how we avoid fragmented traces across independently deployed agents.

7. What Should We Measure?

Collecting telemetry is not enough.

We need to define which metrics are useful for understanding the health and quality of an agent platform.

I would divide AI observability metrics into five categories.

7.1. Latency Metrics

These describe how long operations take.

For streaming applications, time to first token is particularly useful.

A workflow that takes 20 seconds but starts responding after 300 ms may provide a better experience than one that takes 10 seconds and remains silent for the first 8 seconds.

7.2. Reliability Metrics

These help us understand whether workflows complete successfully.

Examples:

• Agent execution success rate
• Tool execution failure rate
• Model request failure rate
• Timeout rate
• Retry rate
• Task cancellation rate
• Human intervention rate
• Incomplete workflow rate

A high retry rate can be an early indicator of a problematic integration or an unstable external dependency.

We should also distinguish between a successful model response and a successful business workflow.

An LLM can return a perfectly valid response while the overall task fails to achieve its objective.

7.3. Token and Model Usage

Token usage should be tracked at multiple levels.

For multi-step workflows, we should aggregate usage from all model calls rather than looking only at the final response.

7.4. Agent and Tool Behavior

These metrics describe how efficiently agents perform their work.

Examples:

• Number of tool calls per task
• Number of model calls per agent invocation
• Number of agent handoffs
• Repeated tool-call frequency
• Tool-selection success
• Average steps per workflow
• Average number of clarification requests

An agent repeatedly calling the same tool can indicate a reasoning loop, an incorrect tool result, or an orchestration problem.

The trace helps us identify the execution pattern. Metrics help us understand how frequently the pattern occurs.

7.5. Quality Metrics

This is where AI observability starts to overlap with AI evaluation.

Examples:

• Answer correctness
• Groundedness
• Retrieval relevance
• Tool-selection accuracy
• Task completion
• Policy adherence
• Human acceptance rate

These metrics cannot always be derived from execution telemetry alone.

We need an evaluation strategy to measure the actual quality of the result.

8. Cost Control: Understanding the Real Cost of an Agent

One of the biggest mistakes in AI platforms is measuring only model token usage and treating it as the total cost of the workflow.

An agent-based application may consume resources across several layers:

• LLM inference
• Embedding generation
• Reranking
• External API calls
• Database queries
• Vector database operations
• Container or sandbox execution
• Remote agent invocations
• Infrastructure and networking

The total cost should reflect the entire execution path.

A useful starting point is to define a model cost formula:

[
C_{\text{model}} =
\sum_{i=1}^{n}
\left(
T_{\text{input},i}P_{\text{input},i}
+
T_{\text{output},i}P_{\text{output},i}
\right)
]

Where:

• (T) represents token usage.
• (P) represents the applicable price per token.
• (i) represents an individual model operation.

The pricing model may also need separate rates for cached input, audio, images, reasoning tokens, or other modalities.

We should use the actual provider-reported billable usage wherever available and avoid double-counting cached or reasoning token subsets.

The complete workflow cost can then be represented as:

[
C_{\text{workflow}} =
C_{\text{model}}
+
C_{\text{tools}}
+
C_{\text{compute}}
+
C_{\text{storage}}
]

This may be an estimate rather than an exact invoice calculation, depending on which provider and infrastructure usage data are available.

Why is cost per successful task more useful?

Imagine two implementations:

Agent A is cheaper per attempt, but Agent B completes more tasks successfully.

Neither is automatically better.

The correct decision depends on the business value of successful completion, quality, latency, and the cost of failed attempts.

A useful metric is:

[
\text{Cost per successful task} =
\frac{\text{Total execution cost}}
{\text{Number of successful tasks}}
]

The numerator should include failed attempts that consumed resources, rather than counting only successful runs.

Model selection and routing

Observability data can also help us make better model-routing decisions.

For example:

• Use a smaller model for classification or simple extraction.
• Use a more capable model for complex analysis.
• Avoid unnecessary model calls when a deterministic operation is sufficient.
• Reuse cached context when appropriate.
• Set explicit iteration and tool-call limits.
• Stop an agent when a task exceeds its budget.

The important principle is that model selection should be based on measured task quality and total execution cost, not just the price of an individual inference request.

Token metrics in OpenTelemetry

The current GenAI token metric conventions include proposed counters such as:

• gen_ai.client.inference.usage.input_tokens
• gen_ai.client.inference.usage.output_tokens
• gen_ai.client.inference.usage.cache_read.input_tokens
• gen_ai.client.inference.usage.reasoning.output_tokens

These conventions are still in Development.

There is also an important distinction between token usage attributes on individual spans and cumulative token usage metrics.

A span attribute describes an individual operation. A counter or histogram is used to analyze consumption across many operations.

For production cost reporting, maintain an application-level pricing registry with model and provider versions, effective dates, and the relevant unit prices.

This makes historical cost calculations more reliable when provider pricing changes.

References: GenAI Token Metrics · GenAI Metrics

9. Observability Is Not Evaluation

This distinction is worth emphasizing.

Observability tells us what happened during execution.

Evaluation tells us whether the result was good enough.

A trace can show that an agent:

• Called the correct model.
• Used three tools.
• Retrieved five documents.
• Completed within four seconds.
• Returned a response without an exception.

But none of these facts proves that the answer was correct.

Consider a RAG application.

The retrieval operation may complete successfully, and the model may return a fluent answer. However, the retrieved documents may be irrelevant or the final answer may contain unsupported information.

From an operational perspective, the request succeeded.

From a quality perspective, it failed.

We need both observability and evaluation.

Offline Evaluation

Offline evaluation measures an AI application against a predefined dataset.

For example, we can create 100 representative questions and expected outcomes.

Whenever we modify a prompt, model, retrieval strategy, or agent workflow, we run the same dataset again.

This helps identify regressions before deploying changes.

Useful evaluation dimensions include:

• Answer correctness
• Relevance
• Groundedness
• Tool selection
• Expected task completion
• Policy compliance

Online Evaluation

Online evaluation measures actual production interactions.

We can select a sample of production traces and evaluate them using:

• Rule-based checks
• Code-based evaluators
• LLM-as-a-Judge
• Human feedback

The important part is to turn real production failures into future test cases.

For example:

1. A production trace shows that an agent selected the wrong tool.
2. We identify the relevant input and expected behavior.
3. We create a regression dataset entry.
4. We update the prompt or agent implementation.
5. We rerun offline evaluations.
6. We deploy and monitor the next set of production traces.

This creates a continuous improvement loop.

Production Traces | v Failure Analysis | v Evaluation Dataset | v Offline Evaluation | v Prompt / Model / Agent Changes | v Deployment | v Online Evaluation | +------> New Regression Cases 

Tools worth evaluating

Two examples of AI-focused observability and evaluation platforms are:

Langfuse

Provides tracing, evaluations, datasets, experiments, and online evaluation of production traffic. It can help connect execution traces with quality measurements.

Documentation: https://langfuse.com/docs/evaluation/overview

Arize Phoenix

An open-source observability and evaluation platform built around OpenTelemetry-based instrumentation and OpenInference conventions.

It provides tracing, evaluation, datasets, experiments, and debugging capabilities.

Documentation: https://phoenix.arize.com/

These tools complement OpenTelemetry. They do not replace the underlying need for correct instrumentation or a well-defined evaluation strategy.

Reference: Langfuse Evaluation Concepts · Arize Phoenix

10. The Role of OpenTelemetry Collector

In a production architecture, applications should not necessarily export telemetry directly to every observability backend.

This is where the OpenTelemetry Collector becomes useful.

The Collector receives telemetry from instrumented applications, processes it, and exports it to one or more destinations.

It provides a central place for:

• Batching telemetry
• Filtering unwanted data
• Applying processors
• Handling authentication
• Reducing telemetry volume
• Sampling traces
• Routing signals to different backends

A simplified architecture looks like this:

 AI Applications | +----------+----------+ | | | v v v Next.js .NET API Python Agent | | | +----------+----------+ | OTLP | v OpenTelemetry Collector | +--------+--------+ | | | v v v Traces Metrics Logs | | | v v v Jaeger Prometheus Log Backend | v Grafana 

The same telemetry can also be routed to AI-specific observability platforms where supported.

Why centralize collection?

In enterprise environments, we may have several applications and teams using different frameworks and languages.

A Collector provides a common ingestion point and makes it possible to apply centralized policies.

For example, a Collector can filter sensitive attributes before telemetry reaches the final backend.

It also allows us to change exporters or observability providers without changing the instrumentation in every application.

Agent and gateway deployment

OpenTelemetry supports different Collector deployment patterns.

A common enterprise approach is to run lightweight Collectors close to applications and forward telemetry to centralized gateway Collectors.

This provides a separation between local telemetry collection and centralized processing.

However, it introduces operational complexity and additional infrastructure. The Collector itself becomes another production component that needs capacity planning, monitoring, and failure handling.

Reference: OpenTelemetry Collector · Gateway Deployment Pattern

11. Sampling: How Do We Control Telemetry Volume?

Agent-based workflows can generate a significant amount of telemetry.

Consider an agent that performs 15 model calls and 20 tool calls during a single task.

If the application handles thousands of tasks per hour, storing every span indefinitely may become expensive.

Sampling helps control this volume.

There are two common approaches.

Head-Based Sampling

The sampling decision is made when the trace starts.

For example, we may decide to sample 10% of incoming requests.

This is relatively simple and reduces the amount of telemetry generated or exported.

However, we do not yet know whether a trace will eventually fail or become unusually slow.

Tail-Based Sampling

The sampling decision is made after considering the trace's collected spans.

This makes it possible to retain:

• All failed traces.
• All traces exceeding a latency threshold.
• Traces containing specific tool failures.
• A percentage of successful traces.

For AI agents, tail-based sampling can be particularly useful because many execution problems only become visible after several operations have completed.

However, tail sampling has an important infrastructure requirement: all spans belonging to the same trace must reach the same tail-sampling Collector instance.

This generally requires trace-ID-aware routing when using multiple Collector instances.

A practical approach is to combine sampling strategies:

• Preserve error and high-latency traces.
• Retain a representative sample of successful traces.
• Maintain complete aggregate metrics where possible.
• Apply stricter retention policies to verbose telemetry.

Reference: OpenTelemetry Sampling · Tail Sampling Processor

12. Security and Privacy Considerations

This is one of the most important aspects of AI observability in enterprise environments.

Traditional telemetry usually contains operational information such as request duration, HTTP status codes, and service names.

AI telemetry can contain much more sensitive information.

For example:

• User prompts
• System instructions
• Generated responses
• Tool arguments
• Tool results
• Retrieved documents
• Conversation context
• Customer or business identifiers

Sending all this information to an observability backend can create additional privacy and data governance risks.

Do not capture prompts and responses by default

A particularly important detail is that the AI SDK OpenTelemetry integration records inputs and outputs by default unless configured otherwise.

We can disable this behavior:

registerTelemetry( new OpenTelemetry({ recordInputs: false, recordOutputs: false, }), ); 

For many production applications, this is a better starting point.

We can still collect useful operational telemetry:

• Model identifiers
• Token counts
• Duration
• Finish reasons
• Tool execution times
• Error information
• Workflow identifiers

without exporting the complete conversation.

Apply data minimization

Before exporting telemetry, ask whether each attribute is genuinely needed.

Avoid sending:

• Access tokens
• Authorization headers
• Passwords
• Raw customer records
• Full database query results
• Unfiltered tool arguments

For sensitive workflows, use approved metadata rather than full payloads.

Separate operational telemetry from application data

An observability backend should not automatically become another storage system for business data.

The fact that a trace is useful for debugging does not mean the full underlying request and response should be retained indefinitely.

Define access and retention policies

Enterprise environments should define:

• Who can access traces.
• Which teams can view prompt and response content.
• How long telemetry is retained.
• How data is deleted.
• Which environments are allowed to capture full content.
• How telemetry exports are authenticated and encrypted.

A sensible approach is to use more detailed content capture in controlled development environments and much stricter policies in production.

Reference: OpenTelem...