What an AI support chatbot actually costs per conversation

"How much will this cost to run?" is the question I get before any technical one, and the honest first answer is annoying: less than you fear, in a shape you did not expect.

People budget for the indexing — the "training on your content" part — because that is the part that sounds expensive. It is the cheapest line on the bill. The money goes somewhere else entirely, and if you do not know where, you will optimise the wrong thing for a month.

Here is the actual shape of it, with a method you can run on your own numbers. I am deliberately not quoting per-token prices: they change every few months, and a blog post with 2026 prices in it will be wrong by the time you read it. Go to your provider's pricing page, take today's numbers, and substitute them into the formulas below.

The four lines on the bill

Every answer your bot gives costs you four things, in wildly different proportions:

1. Indexing — embedding your content. Paid once per change.
2. Query embedding — turning the customer's question into a vector. Paid per question.
3. Generation — the LLM writing the answer. Paid per question, by input and output tokens.
4. Everything that is not AI — hosting, the vector store, bandwidth, your time.

Let me take them in order of how much people over- and under-estimate them.

1. Indexing is rounding error

Embedding is priced per token and embedding models are the cheapest thing in the stack, typically one to two orders of magnitude cheaper per token than a generation model's input.

A mid-sized business site — say 200 pages averaging 800 words — is roughly 210,000 tokens. Embedding that whole site, from scratch, costs a few cents at any current embedding price. Even re-embedding it nightly for a year is less than a single month of generation on a modestly busy bot.

The practical conclusion is a design decision, not a saving: stop treating re-indexing as expensive. Re-crawl nightly. Re-index the moment a page changes. The reason people's bots quote last year's prices is almost never cost — it is that nobody built the refresh.

2. Query embedding is also rounding error

A customer's question is 10–30 tokens. Embedding it costs a fraction of a fraction of a cent. You can safely ignore this line. I mention it only because people sometimes try to cache query embeddings, which is effort spent saving nothing.

3. Generation is the whole bill

Here is where it all is, and the number that drives it is not what people expect. It is not the answer length. It is the context you send.

A typical RAG request looks like:

So you are paying for roughly 4,500 input tokens and 200 output tokens per message. Input tokens are cheaper than output per token, but there are twenty times more of them — on most models' pricing, input is still the majority of the cost of a RAG answer.

And a "conversation" is not one message. Real support conversations run 3–6 messages. So:

cost per conversation = turns × (input_tokens × input_price + output_tokens × output_price) 

Plug in your provider's current per-million-token prices and you have your number. Do this before you price your product, not after.

Why history is the sneaky one

Notice what happens as a conversation goes on. Turn 1 sends 4,500 tokens. Turn 2 sends those 4,500 plus turn 1's question and answer. Turn 5 can be sending 8,000. Naive conversation handling makes cost grow quadratically with conversation length.

Two fixes, both cheap:

// 1. Cap history by tokens, not by message count, and always keep the // first user message — it usually holds the actual intent. function trimHistory(messages, maxTokens = 1200) { const first = messages[0]; const rest = []; let used = estimateTokens(first); for (const m of messages.slice(1).reverse()) { const t = estimateTokens(m); if (used + t > maxTokens) break; rest.unshift(m); used += t; } return [first, ...rest]; } 

1. Re-retrieve per turn instead of accumulating chunks. If turn 3 is about shipping, you do not need turn 1's pricing chunks still sitting in the prompt. Retrieve fresh for each question and the input size stays flat instead of climbing.

The three levers that actually move the number

Prompt caching. Most providers now let you mark a stable prefix — your system prompt, your few-shot examples — as cacheable, and charge much less for cached input tokens on subsequent calls. Structure your prompt so everything stable comes first and everything variable comes last, or the cache never hits. This is the single largest saving available and it costs you one afternoon.

Fewer, better chunks. Going from 10 chunks to 5 halves the largest part of your input. Re-ranking makes this safe — the five you keep are better than the ten you had. Cheaper and more accurate is a rare combination; take it.

A small model for small jobs. Not every call needs your best model. Classifying "is this a greeting, a question, or abuse?", rewriting a follow-up into a standalone query, deciding whether to escalate — these run fine on the cheapest model in the family. Route by job, not by habit.

What does not move the number much: shortening your system prompt by a few sentences, trimming the answer length, or switching embedding models. People spend days there.

4. The costs that are not tokens

For a small deployment these often exceed the AI bill, and they get forgotten in every estimate I have seen:

• The vector store. Hosted vector databases charge for storage and for the pods or units that keep the index warm, and that is a monthly floor whether or not anyone chats. At the scale of a single business's website, pgvector on a Postgres you already run, or SQLite with a vector extension, is genuinely enough — a few hundred thousand vectors is not big data. A managed vector service is a decision to make at scale, not at launch.
• Egress and hosting for the widget, the admin, the webhooks.
• WhatsApp, if you use it, is billed by Meta on its own per-message schedule, entirely separately from your AI costs. It is quite capable of being the largest line on the bill.
• Your time. Thirty minutes a month reading the unanswered-questions log costs more than the tokens.

Two numbers worth writing down

Cost per conversation tells you whether the unit economics work. Compute it from the formula above and sanity-check it against a real week of traffic.

Cost per resolved conversation tells you whether the thing is worth running at all. If 40% of conversations end in a human being involved anyway, your real cost per deflected ticket is 2.5× the headline — and you should compare that against what the ticket would have cost in staff time.

That second number is the one to put in front of whoever approves the spend. It is also the one that improves when you fix your content, which brings everything back to the same place: the bot's economics are mostly a property of how well your website answers questions.

The useful summary

• Indexing: ignore it, and re-index far more often than you do.
• Generation: all of the cost, and most of it is input tokens, not output.
• Conversation history: the thing that quietly makes turn 5 twice the price of turn 1.
• Prompt caching, fewer chunks, cheaper model for small jobs: the three levers that matter.
• The vector store's monthly floor can beat your token bill at small scale. Start with Postgres.

Work out the per-conversation number before you quote a client a monthly price. It takes twenty minutes and it is the difference between a margin and a surprise.

I build ChatNory, an AI chatbot that answers from a business's own content on its website, WhatsApp and Telegram. These numbers come from running it, including the month I spent optimising the wrong line. If your cost shape looks different, I would like to hear why — especially if you are at a scale where the managed vector store starts paying for itself.