With Microsoft.Extensions.AI, switching from a hosted model to a local Ollama server really is close to a one-line change: register a different IChatClient and you're done. The code compiles, the first request works, and you move on.

What doesn't carry over are the assumptions. A hosted API is a big shared service with generous limits that tells you when something is wrong. A local Ollama server is one process, usually running one request at a time, with its own defaults, and it's quiet about its problems.

Here are the five assumptions that bit me while building a local-LLM chat API in C#, how each one showed up, and the shape of the fix.

Tech Skill Builder is $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10. The complete, tested project behind this post (a .NET 10 local-LLM chat API with Clean Architecture and 187 passing tests, running offline against an Ollama simulator) plus a step-by-step PDF guide is inside. Join Tech Skill Builder

A note on honesty before we start: my numbers below come from an in-process simulator of Ollama's HTTP API sitting behind the real OllamaSharp client, modelled on the Ollama FAQ and public issue reports. I didn't run a real model or a GPU for them. Treat them as illustrations, and measure your own setup.

Mistake 1: "If it returned 200, the model saw my prompt"

With a hosted API, a prompt that's too long gets rejected with a clear error. Ollama's documented default context window is 4096 tokens, and public issue threads describe the server cutting an oversized prompt, logging a warning on its own side, and answering normally with a 200. The oldest content goes first, which can include your system prompt.

How it shows up: confident, wrong answers about things that were clearly in the input. In my simulator run with default settings, 15 of 29 long-document questions came back with a 200 and a wrong answer.

The fix, in shape: the response has no "truncated" flag, but it does report prompt_eval_count. Compare it to what you sent and to the window:

public TruncationSigns Inspect(int estimatedPrompt, int reportedPrompt, int outputTokens, int contextWindow) { var signs = TruncationSigns.None; if (reportedPrompt >= contextWindow * 0.97) signs |= TruncationSigns.PromptAtLimit; // ...two more checks (prompt shrank, window exhausted) // full implementation in the complete project return signs; }

It's a heuristic, not proof (a prompt cache can lower the count too), so I surface it as a warning by default.

Mistake 2: "The context window is a property of the model"

It's a property of the request. If you don't send num_ctx, you get the server's default, whatever the model could handle. ChatOptions has no typed property for it, so it's easy to never set it.

The fix, in shape: the gateway sends the window and the keep-alive on every request, and a planner decides what fits before the call, instead of letting the server decide what to throw away:

public ContextPlan Plan(Message system, IReadOnlyList<Message> history, Message newest, int limit) { // 1. system + newest must fit, otherwise refuse (413), send nothing // 2. drop the oldest turns in question/answer pairs until it fits // 3. estimate pessimistically, learn from reported token counts // full implementation in the complete project }

The trade-off is real: a pessimistic estimate occasionally refuses a prompt that would have fit. I'd rather get a clear 413 than a quietly wrong answer.

Mistake 3: "Concurrency is the server's problem"

OLLAMA_NUM_PARALLEL defaults to 1. Everything else waits in a server-side queue that you can't see from your app, which defaults to 512 places. Under a burst, your users just wait, and wait.

How it shows up: latency that climbs with load and never errors. In the simulator, 30 simultaneous requests gave a p95 of 3,311 ms.

The fix, in shape: put a small, visible gate in front of the model: as many slots as the server has, a short queue, a maximum wait, and a fast 429 with Retry-After beyond that.

public async Task<GateEntry> EnterAsync(CancellationToken ct) { // free slot -> run now // queue full -> QueueFull immediately (maps to 429) // waited longer than MaxWait -> WaitedTooLong (maps to 429) // full implementation in the complete project }

With a queue of 4 and a 1-second wait, 5 requests were accepted at a p95 of 626 ms and 25 got a quick 429. A short line with a fast "no" beats a long line with a slow "yes".

Mistake 4: "The default resilience handler is fine"

AddStandardResilienceHandler gives each attempt 10 seconds and retries on timeouts. Both defaults hurt here. A non-streaming chat returns headers only when the whole answer is done, and a cold load of a bigger model can take longer than 10 seconds by itself. Retrying a timed-out request usually means the server is still computing the first one, so you've just doubled the load.

The fix, in shape: set the attempt timeout to the real time-to-finish you're willing to accept, size the total timeout from it, and retry only on connection failures and 503, never on timeouts. One gotcha: the standard handler won't accept zero retries.

http.AddStandardResilienceHandler(r => { r.AttemptTimeout.Timeout = options.RequestTimeout; r.Retry.ShouldHandle = args => ValueTask.FromResult(IsTransient(args.Outcome)); // total timeout sizing + IsTransient: full implementation in the complete project });

Mistake 5: "Healthy means ready"

The process being up doesn't mean the model is in memory. The first request after a start, or after the default five idle minutes, pays for the load. In the simulator, that first request took 1,693 ms cold versus 96 ms warm.

The fix, in shape: a hosted worker sends an empty chat request at startup (the Ollama FAQ documents this as a way to preload a model), keeps the model alive, and /health/ready returns 503 until the model is actually loaded. Your load balancer then never sends a real user into a cold start.

Bonus: when the line gets long, get faster instead of longer

The project also has a brownout: when the queue-wait p95 crosses a threshold, requests switch to a smaller fallback model, with separate enter and exit thresholds so it doesn't flap. In the simulator, a steady stream of 60 requests went from a p95 of 3,042 ms to 415 ms. That's a quality trade, so it's a decision you make on purpose, not a default.

Quick checklist

  • [ ] num_ctx and keep_alive are sent on every request.
  • [ ] Context is planned before the call; oversized input gets a clear 413.
  • [ ] prompt_eval_count is compared with what you sent.
  • [ ] A bounded gate matches OLLAMA_NUM_PARALLEL and sheds load with 429.
  • [ ] Timeouts fit a cold load; timeouts are never retried.
  • [ ] Readiness means "model loaded", not "process running".
  • [ ] Ollama stays on localhost, behind your own auth.

Get the complete version

This post gives you the map and the skeletons. The complete project fills them in: the context planner with its self-correcting token estimator, the truncation detector with an opt-in strict mode, the bounded gate, the resilience setup, the warm-up worker and readiness check, the brownout controller, a streaming endpoint with separate first-token and total budgets, metrics, and the tests that prove each piece.

The complete, working, tested project (.NET 10, Clean Architecture, 187 passing tests: 156 unit + 31 integration, runs offline against an Ollama simulator, no model or GPU needed, with a compose file for a real Ollama) plus a step-by-step PDF guide is in Tech Skill Builder. It's $5/month for a very limited time only (normally $10). Grab it now before it goes back to $10: https://elitesolutions1.gumroad.com/l/TechSkillBuilder

Have you moved anything from a hosted model to a local one? I'm curious which of these surprised you, or what broke first for you. Tell me in the comments.