If you have an AI engineer onsite coming up, the round to worry about is system design. It runs 45 to 60 minutes, it carries the most weight, and it's where people who breezed through the coding screen lose the offer.
The good news is that the prompts barely change. We went through reported AI system design questions from 2026 interview loops, and the same ten kept coming back. Design ChatGPT. Design a RAG system. Design a support bot. Design a banking chatbot that doesn't hallucinate. A few more.
They look like ten different problems. They get graded as one. The interviewer isn't checking whether you memorised a ChatGPT architecture. They're watching whether you can walk the same seven steps under a clock and put a number on each.
So this post gives you the seven steps first, then the ten prompts with the tradeoff each one is really testing. The full worked answers, with diagrams, latency budgets and cost math, are in a free repo we maintain: https://github.com/landedjobs/awesome-ai-engineer-interview
The seven steps every answer walks
Chip Huyen's book, HelloInterview's rubric and the open templates on GitHub all land on the same spine. What separates candidates is walking it without skipping and saying a number at every step.
1. Frame the problem. Ask about scale, freshness, tenancy, stakes and latency. Then say the numbers you're assuming out loud and get them confirmed. "Ten million requests a day is about 116 a second on average, call it 350 at peak. I'll assume a p99 of 500 ms. Does that match?" That sentence does more for you than any diagram.
2. Design the eval set before the system. LLM apps rarely have clean labels. Say where the quality signal will come from, and propose 50 to 200 hand-written cases (question, expected answer, expected source) that you grow from production traces.
3. Pick a baseline, then the simplest thing that beats it. BM25 for search. One prompt for generation. A router with canned answers for support. Without a baseline you can't say what your complexity buys.
4. Budget latency per stage. "It'll be under 500 ms" loses points. "Embed 15, search 20, rerank 60, first token 300, guardrails 40, the rest is buffer" wins them. For LLMs, split time to first token from time between tokens. The first is compute-bound prefill, the second is memory-bandwidth-bound decode, and they have different fixes.
5. Pair every metric with a counter-metric. Deflection rate with re-opened tickets. Relevance with false refusals. Offline scores are a gate. The A/B test is the verdict.
6. Put a dollar figure on it. Input tokens times input price, plus output tokens times output price, and output usually costs 3 to 5 times more per token. Multiply out to a month.
7. Say what breaks at 10x. The GPU queue, the index RAM, the cold start. Then how you degrade instead of falling over, like serving a cheaper model instead of timing out.
Watch the clock as closely as the content. In a 60 minute round, finish framing and the eval set by minute 18. Completing all seven steps beats a perfect step three. Most people who fail this round fail it in the first ten minutes, by drawing a vector database before anyone has agreed what "good" means.
The 10 prompts, and what each one is really testing
1. Design ChatGPT
This is a serving question wearing a product costume. The graded tradeoff is latency against cost at scale. Talk about continuous batching, PagedAttention (the vLLM paper cut wasted KV-cache memory from 60-80% to under 4%), streaming, where conversation state lives, and prefix caching on the system prompt. The usual miss is giving one latency number instead of two.
2. Design a RAG system
Scope it before you build it. How many documents, how fresh, and who is allowed to see what? Then ingestion, hybrid retrieval (dense plus BM25, fused with reciprocal rank fusion), a reranker, and a cited answer. Do the RAM math on the board. 50 million chunks at 1,536 dimensions and 4 bytes each is about 300 GB before index overhead, so it won't sit on one machine and you'll shard or compress. If you want a strong number, Anthropic's contextual retrieval write-up cut failed retrievals by 49% with contextual embeddings plus BM25, and 67% once a reranker was added.
3. Design LLM inference at scale
GPU economics with a latency mask. Start with memory. A 70B model at FP16 is 140 GB, which doesn't fit on one 80 GB H100. FP8 halves that and int4 quarters it, if a quality check says the output holds up. Then the KV cache, which is the real memory pressure under load. Autoscale on queue depth, not GPU memory, because vLLM preallocates memory and the graph will always look full. The trap answer is "buy bigger GPUs to make tokens stream faster". Decode speed is bound by memory bandwidth, so a bigger card mostly helps prefill and concurrency.
4. Design an agent
Start by arguing for the simplest pattern that works, usually a single loop. Then the math that drives everything else. If each step succeeds 95% of the time, a 10-step task succeeds about 60% of the time and a 20-step task about 36%. So the design is really about catching and recovering from failure. Fewer than 20 tools. Typed errors that say "retry", "try another way" or "ask a human". State saved outside the model. Traces you can replay. Grade it on pass^k (all k attempts succeed), not pass@k, which flatters you.
5. Design an AI tutoring platform
This one has been reported at Oracle. The twist is that a correct answer isn't the goal. The student learning is the goal, so your eval has to measure teaching quality as well as faithfulness. Add safety rules for minors and cost at classroom scale. The tradeoff to name is grounding answers in the curriculum against being a helpful conversation partner.
6. Design a Claude chat service
Same serving spine as ChatGPT, with the weight moved. Spend longer on safety checks for both inputs and outputs, on long context (when to retrieve instead of stuffing 200k tokens into every call), on tool use through MCP, and on keeping tenants isolated. Prepare this variant for frontier labs.
7. Design a customer-support chatbot with RAG
The business metric is deflection with customer satisfaction held steady. Deflecting with wrong answers is worse than escalating. The shape is a router that picks between answering, acting and escalating, RAG over the help centre, and a short list of typed tools like get_order and issue_refund, with a human approving anything that moves money. The baseline is an intent classifier with canned replies for the top 30 intents. Mention that cost grows with every turn, because each turn resends the transcript. The worked answer lands around 3 cents a conversation.
8. Design a real-time chatbot API
A service-level question. Streaming, budgets for first token and per token, rate limits with backoff, hedged requests for tail latency, autoscaling, and a fallback to a smaller model. The senior detail is that the fallback needs its own eval, because users get the degraded answer too.
9. Design a hallucination-free banking chatbot
Push back on the premise first, politely. No LLM is provably hallucination-free. What you can build is grounded-or-refuse. Route transactional and out-of-scope requests away from free-form generation. Never let the model produce a number. It calls a typed balance API and quotes the result. Check every claim against a cited source before the answer goes out. Build three eval slices: questions it must answer, questions it must refuse, and adversarial ones. And track the false-refusal rate, because a bot that refuses everything is broken in a different way.
10. Design candidate sourcing over 750M profiles in under 500 ms
Pure retrieval at scale, and the RAM math decides the design. 750 million profiles at 1,024 dimensions and 4 bytes each is about 3 TB raw. HNSW won't fit comfortably, so you compress with IVF-PQ (roughly 20x, to about 150 GB) and shard. Apply filters like location, seniority, recruiter permissions and data residency before the vector search, because filtering afterwards returns fewer than k results on narrow queries. Use two stages: vector search down to 500, a ranker down to 25. And in hiring, a fairness check on the ranked list is a legal requirement, not a nice-to-have.
What a pass sounds like
Three habits show up in almost every strong answer. The candidate names a baseline by minute 18. They describe quality in business terms, like deflection rate or the cost of one wrong answer, instead of model loss. And they bring up a counter-metric before anyone asks.
The fourth habit is the one people skip. Check in after each step: "Does this match what you're looking for, or should I go deeper here?" In close calls, clear communication beats raw depth.
To practise, pick three of these prompts, set a 45 minute timer, and answer out loud. Record it. Then compare it with the worked design and find the steps where you went quiet.
Get the full set
The repo has the rubric, ten worked designs with latency budgets and monthly cost estimates, 267 interview questions with answers, and breakdowns of the loops at 12 companies including OpenAI, Anthropic and Google DeepMind. It's MIT licensed and free. If it saves you a weekend, a star helps the next person find it.
https://github.com/landedjobs/awesome-ai-engineer-interview
If you had an AI system design round this year, what was the prompt? I'd like to add the ones that aren't on this list yet.
I'm building landed, which finds jobs that fit you and applies to them early.
