Every store owner who adds an AI chat widget asks the same thing a month later: "Is it actually selling anything, or is it just a cost?" Chat counts and thumbs-up ratings don't answer that. Orders do.

I run the tech for a bookstore that sells in 4 countries. When we added an AI assistant, I made one rule before writing any prompt: the bot only stays if it can show which orders it created.

Tagging the sale at the source

The assistant recommends books as cards with an Add to Cart button. That button calls Shopify's /cart/add.js with one extra field:

await fetch('/cart/add.js', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ items: [{ id: variantId, quantity: 1, properties: { _ds_chatbot: '1' } }] }) });

The underscore prefix keeps the property hidden from the customer in cart and checkout, but it stays on the order forever. A reporting script then pulls orders through the Admin GraphQL API, 25 orders × 25 line items per page so it stays under the cost limit. It splits them into:

  • assisted revenue: the full order value when any item came from the bot
  • direct revenue: only the lines the bot added itself

That gives two honest numbers instead of one inflated one.

The limit I tell owners about up front

This counts only purchases made through the bot's own button. A customer who chats, closes the widget and finds the book by hand isn't counted. So the real impact is at least what the report shows, never less. The next step is a session-level cart attribute, but I'd rather under-report than guess.

Three things that broke in production

1. All three AI providers ran out on the same day. The first version spread traffic across cheap providers. One hit its 100k-token daily cap, the backup had zero credits, and a third account was empty too. The bot went quiet while customers were still writing. We moved to a single Claude API account with funded billing. Fewer moving parts beat cheaper parts.

2. "5 se 15 pounds" returned a £48.60 book. Customers write in English, Urdu and Roman Urdu, often mixed. Every budget and topic rule was an English regex, so a Roman Urdu price range fell straight through. I replaced the regexes with a small classifier call that runs before the reply. It's forced to return structured data:

{ "topicHandles": ["children"], "priceMin": 5, "priceMax": 15, "needsBudgetAsk": false }

It reads the last 10 turns, so "isi range me dekha do" ("show me more in this range") keeps the budget from three messages ago.

3. Prompt caching silently did nothing. I added cache markers and got cache_creation_input_tokens: 0, with no error. The model's minimum cacheable prefix was 4,096 tokens and my static prompt was about 3,400. Moving the whole knowledge base into the cached block brought it to 4,661 and caching started working. When caching silently fails, check the minimum prefix size before anything else.

What it costs

Each customer message makes 2 calls: the classifier and the reply. With most of the prompt served from cache:

  • 1,000 messages/month ≈ $3–4
  • 5,000 messages/month ≈ $15–20

Compare that against assisted revenue, and "is the bot worth it" stops being a matter of opinion.

The takeaway

An AI feature on a store is a sales channel, so measure it like one. Tag the sale where it happens, report the conservative number, and design for the provider going down.

I write about building real systems for Shopify stores at fahadazhar.site. If you're adding AI to a store and want to know whether it pays for itself, happy to compare notes in the comments.