Scaling a normal API is a solved problem: make the servers stateless, put the state in a database, add nodes. An AI chat backend follows the same recipe and still goes wrong, because a chat turn is slow, costs money, depends on history, and gets retried by clients all the time.
When I moved a C# chat API from one node to three, nothing crashed. Every response was a 200. And the bill and the answers were both quietly wrong.
So instead of a tutorial, here are the six questions I now ask before a chat backend gets a second node, with the shape of the answer to each.
Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. The complete, tested project behind this post (a .NET 10 multi-node chat backend with Clean Architecture and 198 passing tests, running offline against a simulated model) plus a step-by-step PDF guide is inside. Join Tech Skill Builder
About the numbers: they come from my own local runs, with real Kestrel processes on one Linux box sharing one SQLite file and a simulated model at about 400 ms. They show how the backend behaves, not how fast any model is.
Question 1: "Can my server tell a retry from a new message?"
If the answer is "no", every timeout costs you a second model call. Phones lose signal, browsers time out, users tap Send twice, and the retry lands on whatever node the load balancer picks.
In my test, 60 messages each retried on another node produced 120 model calls and 60 duplicate messages in the history.
The shape of the answer: the client creates a clientMessageId when the user presses Send and repeats it on every retry. The server checks it in the shared store before anything else:
public sealed record SendMessage( Guid ConversationId, Guid ClientMessageId, // created by the client, repeated on every retry string Text); // In the admission rules, the retry check comes first: // - same id, same text, already answered -> return the stored reply // - same id, same text, still running -> 202, poll later // - same id, DIFFERENT text -> 422 (a client bug, not a retry) // full implementation in the complete project Storing a hash of the text matters more than it looks. Without it, a buggy client that reuses an id gets the answer to someone else's question.
Question 2: "What happens when the same user has two tabs open?"
Two tabs are two clients. If both send a message while an answer is being generated, both turns read the same history and one of them answers without seeing the other's question.
With two nodes and 240 two-tab messages, I got 115 replies built on stale history.
The shape of the answer: one running turn per conversation, enforced in the store. You still have to choose what the second message gets:
| Option | What the user sees | Cost |
|---|---|---|
Reject with 409 + Retry-After | A short wait, then the message goes through | Client must retry |
| Queue it behind the first | Nothing, just slower | You need a durable queue |
| Cancel the first, answer the newest | The first answer disappears | A paid call thrown away |
I went with 409, and stale replies went from 115 to 0. The price was 120 extra 409 responses that the client retried. There's no free option here, only one you chose on purpose.
Question 3: "Why not sticky sessions, or a lock in memory?"
This is the one that catches most teams, because sticky sessions look like they fix everything above.
They don't. A retry after a crash or a deploy goes to another node anyway. Two tabs may land on different nodes. And a SemaphoreSlim or lock is one lock per process, so with three nodes you have three locks that don't know about each other.
The shape of the answer: correctness lives in the shared store, in one transaction that reads, decides and writes:
public async Task<BeginTurnResult> BeginTurnAsync(BeginTurn command, CancellationToken ct) { // 1. open a write transaction (BEGIN IMMEDIATE in SQLite, SERIALIZABLE in Postgres) // 2. read: conversation, turn for this clientMessageId, active turn, tenant load // 3. decide with a pure Domain function (replay / busy / over cap / start) // 4. write the new turn + a fresh fencing token, then commit // full implementation in the complete project } The decision being a pure function is what makes it testable. In my concurrency test, 24 simultaneous begins from 4 store instances produce exactly one winner every time.
Question 4: "If a node loses its lease, can it still write?"
A lease stops two turns from running at once. It doesn't stop a late one from writing. A node pauses (GC, network blip, a slow model), its lease expires, another node takes over, and then the first one wakes up and saves its old answer on top.
The shape of the answer: every turn gets a fencing token that goes up by one each time ownership changes, and the commit refuses anything that isn't the current token:
public enum CommitStatus { Committed, StaleToken, VersionConflict } // CompleteTurnAsync, in shape: // if turn.FencingToken != command.FencingToken -> StaleToken (someone newer owns it) // if conversation.Version != expected -> VersionConflict (history moved) // otherwise: save user message + reply together, free the conversation // full implementation in the complete project Saving the question and the answer in the same commit has a nice side effect: the history never contains a question without its answer. Any stale-token rejection is also worth an alert, because it means your nodes are pausing longer than the lease.
Question 5: "Who owns the answer when the client disconnects?"
If the model call uses HttpContext.RequestAborted, a client that disconnects cancels the generation. The answer is lost, and the retry pays for it again.
The shape of the answer: the turn runs on its own cancellation token, linked to things that should really stop it (the node shutting down, the lease being lost, a turn timeout), not to the HTTP request:
using var turnCts = CancellationTokenSource.CreateLinkedTokenSource( lifecycle.HardStop, // node really stopping leaseLost.Token); // another attempt took over turnCts.CancelAfter(settings.TurnTimeout); // model call + commit use turnCts.Token / CancellationToken.None, never RequestAborted // full implementation in the complete project Then the retry with the same clientMessageId (Question 1) gets the stored reply in milliseconds. These two ideas only work together.
Question 6: "What happens to in-flight answers during a deploy?"
Rolling deploys send SIGTERM, wait, then kill. Without a plan, every answer a node is generating at that moment is gone.
In my restart test, a hard kill kept 0 of 10 first requests, and the slowest user waited 18.6 seconds. With a graceful drain it kept 10 of 10, and the slowest waited 3.3 seconds.
The shape of the answer: IHostedLifecycleService.StoppingAsync runs before Kestrel stops, so the node can drain while it can still respond:
public async Task StoppingAsync(CancellationToken ct) { // 1. readiness -> 503, new turns -> 503 + Connection: close // 2. short delay so the load balancer notices // 3. wait for in-flight turns to commit, up to a drain timeout // 4. anything left becomes Interrupted (retryable), not stuck until lease expiry // full implementation in the complete project } Don't forget HostOptions.ShutdownTimeout: if it's shorter than your drain, the host cuts the drain short.
A bonus question: "Who owns retries to the model?"
The OpenAI SDK retries by itself. So does AddStandardResilienceHandler(). Stack them and one failure becomes several billed calls. Pick one layer, set the SDK's retries to zero if the handler owns them, and raise the default 10-second attempt timeout, which is too short for a long answer.
The checklist I use now
- Every message carries a client-generated id, checked first in the shared store
- One running turn per conversation, with a deliberate choice for the second tab
- No correctness that depends on sticky sessions or in-memory locks
- Leases plus fencing tokens on the commit
- The turn's cancellation is detached from the HTTP request
- Drain on SIGTERM, with a shutdown timeout longer than the drain
- Exactly one retry layer for model calls
With all of it in place, 3 nodes handled the same load at 49.4 chats per second against 17.3 on one node, with 0 duplicates and 0 stale replies in my runs.
Get the complete version
The snippets above are the skeleton. The full project has the parts that took the longest to get right: the complete admission rules and store transactions, a lease heartbeat that cancels a model call the moment another node takes over, a reaper for turns left by crashed nodes, per-tenant token budgets, a bounded per-node gate that returns 503 instead of queueing forever, and a three-node docker-compose file as a starting point.
The complete, working, tested project (198 tests, Clean Architecture, .NET 10) and a step-by-step PDF guide are in Tech Skill Builder, $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10: Join Tech Skill Builder
Which would you pick for the second tab: reject, queue, or cancel? I'm curious what others went with.