Your AI Feature Is Burning Tokens: A Profiling Checklist for Marketplace Developers

If you have shipped an AI feature on a marketplace, you have probably had this moment: the feature works, users like it, and then the provider invoice arrives looking like it belongs to a much bigger company. Smart search, generated listing descriptions, review summaries and inquiry routing all quietly spend tokens on every request, and marketplace traffic is spiky by nature. A good week for growth becomes a bad week for the AI budget.

The encouraging part is that token waste is rarely spread evenly. It usually hides in a handful of call sites, and the fixes are well known. This post treats token usage the way you would treat a slow endpoint: profile it, find the hot path, then fix the expensive parts in order of impact.

First, understand what you are actually paying for

Every LLM call is billed in two directions. Input tokens are everything you send: system prompt, instructions, retrieved listings, chat history, tool definitions. Output tokens are everything the model writes back.

According to the October 2026 list prices quoted in the original Journeyhorizon guide on reducing token usage for AI marketplace features:

Gemini 3.8 Flash is also listed in the guide at $0.75 and $3.75 per million. Always confirm current prices on the provider pages before you commit to a budget.

Three details catch teams off guard:

• Output is roughly 5x the price of input on these models, so a verbose response costs more than a long prompt.
• Reasoning tokens bill as output. If a model thinks for 2,000 tokens and answers in 100, you pay for 2,100 output tokens.
• The gap between tiers is about 20x. A frontier model can cost around twenty times more per token than the small model from the same provider.

Step 1: Instrument before you optimize

You cannot reduce what you cannot see. Wrap every model call in a shared client that logs, at minimum:

{ "feature": "seller_review_summary", "model": "claude-sonnet-5.5", "input_tokens": 4012, "output_tokens": 288, "cache_read_tokens": 0, "cache_write_tokens": 0, "listing_id": "lst_123", "user_id": "usr_456" } 

With that data you can answer the three questions that matter:

1. Which features consume the most tokens per day?
2. What are the average and p95 token counts per request?
3. What is the cache hit rate?

Then treat tokens like a performance budget. Give each operation a ceiling and fail the release check, or at least fire an alert, when a change pushes it over.

Step 2: Fix the architecture, not just the prompt

Most big wins come from deciding what reaches the model at all.

Send only the context the model needs

Raw listing pages are full of HTML, boilerplate and duplicated fields. Pre-compute a compact summary for each listing and store it. Use retrieval to pick the top candidates first, then send only those. Strip markup before it ever reaches the prompt.

Don't call an LLM for a database query

Sorting by price or distance is a SELECT with an ORDER BY. Keyword and semantic retrieval belong in a search index or vector store. Keep the LLM for the final re-ranking step or for genuinely vague queries such as "something cosy for a rainy weekend".

Route by difficulty

Classification, tagging, short extraction and moderation pre-screening are perfect jobs for small models. Reserve frontier models for complex reasoning, long-form writing or sensitive decisions. The guide points to research results here: RouteLLM reported more than 2x cost cuts in some settings, and FrugalGPT reported up to 98% reduction on its test sets. Your mileage will vary, but the direction is clear.

A simple cascade looks like this:

def classify(listing): result = small_model(listing) if is_valid(result): return result return frontier_model(listing) # escalate only failures 

Just remember to count retries and escalations in your cost math, or the cascade can look cheaper on paper than it is.

Step 3: Tighten the prompts

• Cut filler and repeated instructions. Every sentence you send is billed on every call.
• Use one or two short examples, and put them in the cached part of the prompt.
• Constrain the output. Ask for JSON, set word limits and always set a hard max_tokens.
• Use the lowest reasoning effort that still meets your quality bar.
• Template your prompts with placeholders so you can A/B test them consistently.

For messy inputs you can also try prompt compression. Microsoft Research's LLMLingua reported up to 20x compression with little performance loss in its experiments, but apply it only after you have removed context you didn't need in the first place.

Step 4: Cache and batch aggressively

Provider prompt caching

Both major providers discount repeated prompt prefixes. Per the guide, on Anthropic a cache write costs 1.25x base input with a default five-minute lifetime refreshed on use, and a cache read on Sonnet 5.5 is 0.05x base input, about $0.10 per million tokens; the minimum cacheable length on the 5.5 models is 512 tokens. On OpenAI, caching is automatic for supported models once prompts reach 1,024 tokens on GPT-5.6 and later, with reads at 0.1x the uncached rate (0.05x for gpt-6.1-sol).

The rule that makes caching work: stable prefix first, dynamic content last. System prompt, taxonomy and examples go at the top and never change. The user query, listing data and timestamps go at the bottom. One stray timestamp at the top of the prompt will break every cache hit.

Application-level caching

Generated descriptions, review summaries and tags should be stored and regenerated only when their source data changes. Frequent identical queries can be answered from your own cache.

Batch APIs

Anthropic, OpenAI and Google all offer asynchronous batch processing, usually completed within 24 hours, at a 50% discount, and it can stack with prompt caching. Nightly description generation, catalog re-tagging, moderation backlogs and review summaries are ideal candidates. Packing many items into one prompt can save further on repeated instructions, but it hurts error isolation and can lower quality, so test the pack size against accuracy.

Step 5: Keep conversations and agents on a diet

• Keep only recent turns verbatim and summarize older ones.
• Drop tool results once they have been used.
• Load only the tool definitions the current task needs.
• Cap turns and total tokens per task so an agent can't loop forever.

A worked example: seller review summaries

The guide walks through an illustrative case using October 2026 list prices.

Naive version: 10,000 seller profile views per day. Each view sends about 4,000 input tokens to Sonnet 5.5 and gets a 300-token summary back. That's 40M input tokens ($80) plus 3M output tokens ($30), roughly $110 per day.

Optimized version: Generate summaries once and refresh only sellers who received new reviews. With about 2,000 changed sellers per day, a nightly batch uses 8M input and 0.6M output tokens. With the 50% batch discount that comes to about $11 per day, and on Haiku 5.5 about $0.55 per day.

The trade-off is that summaries refresh daily instead of instantly, which is acceptable for most marketplaces.

When to bother

Optimize when the feature is core, heavily used, margins are thin or users are price-sensitive. Hold off when the feature is experimental or low-traffic: ship something simple, measure it, then optimize. Off-the-shelf options such as an AI search box or AI description generator plugin for Sharetribe are a cheap way to validate demand before a custom build.

The checklist

• [ ] Token counts logged per request, feature and model
• [ ] Context trimmed to required fields and top candidates
• [ ] Stable prompt prefix first, dynamic content last
• [ ] max_tokens and output format set on every call
• [ ] Small models for simple tasks, with escalation rules
• [ ] Batch APIs for anything non-urgent
• [ ] Generated content stored and refreshed only on change
• [ ] Conversation history summarized, agent loops capped
• [ ] Quality measured alongside cost after every change

That last item matters most. Caching and batching don't change outputs, but smaller models and compression can, so run every change against a fixed evaluation set.

Wrapping up

Token reduction isn't a one-off cleanup; it's ongoing design work, just like query performance. If you want a second pair of eyes on your AI architecture, the team at Journeyhorizon offers AI cost optimization for marketplaces, from instrumentation to routing and caching strategy.