Your LLM Costs Are Growing. Here's What to Fix First.

LLM costs can look manageable during development. Then the application goes into production, more users arrive, and one request starts triggering multiple model calls, agents, tools, and retries.

The bill grows, but which part of the application is responsible?

Choosing a cheaper model might help. But the bigger savings often come from reducing unnecessary work across the entire AI workflow.

Here are five places to look.

1. Stop using your most expensive model for everything

Not every task needs advanced reasoning.

A complex planning step might justify a powerful model. Extracting information from a document, classifying a request, or loading context might work just as well with a smaller one.

A practical routing strategy:

For example, one approach discussed in our AI production panel used Anthropic Opus for complex reasoning, Sonnet for intermediate tasks, and Haiku for simpler operations.

But there's a catch.

The cheapest orchestrator isn't always the cheapest workflow.

If a weaker orchestrator sends work to three unnecessary agents, the extra model calls, tool invocations, and retries can cost more than using a stronger model for the initial decision.

Optimize for the cost of completing the task successfully, not just the price of an individual model call.

2. Stop paying to process the same context repeatedly

Look at what's inside your prompts.

System instructions, tool definitions, reference documents, and application rules often stay the same across requests. Yet applications may repeatedly send that context to the model.

Prompt caching can reduce this repeated processing on supported providers.

A useful prompt structure:

[CACHEABLE PREFIX] System instructions Tool definitions Reference context Examples [DYNAMIC CONTENT] Session context Current user request 

Keep stable content together and place frequently changing information later, subject to your provider's caching requirements.

But don't assume caching automatically saves money. Cache writes, reads, minimum token thresholds, and expiration rules vary by provider.

Measure cache usage and compare the actual cost before and after.

See: Amazon Bedrock prompt caching documentation.

3. Find out how many model calls one request actually creates

A user sees one request. Your application might execute something like this:

User request | Orchestrator ($) | +-- Agent A --> Model --> Tool ($) | +-- Agent B --> Model --> Tool ($) | +-- Retry --> Model ($) | Final response ($) 

That single interaction might involve several billable model calls.

Provider billing can tell you how much you spent, but not necessarily which workflow decision caused the expense.

Track token usage and cost by:

• Model and agent
• Workflow and session
• User or service
• Retries and fallbacks

Then investigate which workflows are generating unnecessary calls.

This is where distributed tracing becomes valuable. Connecting model calls, agent steps, and tool invocations within the same trace helps reveal where the cost is coming from.

With OpenObserve's AI integrations, teams can collect telemetry for AI sessions, model calls, tokens, latency, and costs alongside application traces.

4. Watch for hidden cost multipliers

Some of the most expensive AI behavior doesn't look like an error.

That last one is particularly important.

Imagine switching to a cheaper model that reduces inference cost by 40%. If the model produces enough poor responses that users must retry, your actual savings could be much smaller.

A cheaper request is not necessarily a cheaper completed task.

5. Give token usage an owner

Cost optimization becomes difficult when nobody knows which application, team, or workflow is responsible for spending.

Instead of only tracking a monthly total, attribute usage to meaningful owners and set thresholds.

Compare these two situations:

Without attribution: AI spending increased 35% this month.

With attribution: A new agent added three model calls to every execution of one workflow, driving most of the increase.

The second tells you what to investigate.

Budgets don't have to mean restricting experimentation. They make unusually expensive usage visible so teams can decide when additional spending is justified.

What should you optimize first?

Start with this checklist:

• Route simple tasks to smaller models where quality remains acceptable.
• Cache stable prompt context and measure actual savings.
• Trace complete workflows, not just individual model calls.
• Identify unnecessary retries, fallbacks, and agent loops.
• Attribute token usage to workflows, users, or services.
• Compare cost, latency, and task success before and after changes.

The metric that matters: cost per successful task

A useful starting metric is:

Cost per successful task = Total workflow cost / Number of successful tasks

This includes the cost of failed attempts and retries in the total, rather than counting only the final successful model call.

You still need a reliable way to define and measure task success. That may involve automated evaluations, workflow checks, or other task-specific signals.

The goal isn't to minimize tokens at any cost. It's to use the least expensive combination of models, prompts, and agent steps that can reliably complete the work.

For more production lessons on LLM costs, evaluation, and agent observability, watch What Nobody Tells You About Running AI in Production.