The LLM system design interview
An LLM system design interview is a whiteboard design round for a user-facing feature built on a language model: ingestion, retrieval, prompting or orchestration, serving, and the feedback loop. It is a system design interview first — the distributed systems fundamentals do not go away because there is a model in the diagram. The conversation moves fast from the happy path to failure, and the model is rarely the hard part.
For the broader version covering classical ML and hybrid systems, read the AI system design guide. If your round is named AI Design, it may be product-shaped instead; the AI design round guide helps you tell.
What the name usually means
- You are asked to design a user-facing feature built on a language model, end to end: ingestion, retrieval, prompting or orchestration, serving, and the feedback loop.
- The conversation moves quickly from the happy path to failure: what happens when retrieval returns nothing useful, when the model is confidently wrong, when latency triples at peak.
- It is a system design interview first. Queues, idempotency, caching, rate limits and observability are still what most of the score is made of.
- Prompt text is almost never the subject. It is a detail inside the design, and candidates who spend the round on it usually run out of time before evaluation and reliability.
What it tends to test
- Retrieval architecture
- Chunking, indexing, hybrid search, freshness, and what you do when the top-k is junk. Vague retrieval answers are the most common failure in this round.
- Evaluation design
- How you would know the system works before users tell you: an eval set, a metric that moves, and a regression gate that blocks a release.
- Latency and cost judgement
- Token budgets, caching, model sizing, streaming, and when the correct answer is to not call a model at all.
- Failure handling
- Fallbacks, timeouts, partial results, and what the user actually sees when the model is unavailable or wrong.
- Boundaries and data flow
- Where user data goes, what gets logged, what is sent to a third-party provider, and what must not be.
- Context construction
- Treating the window as a budget to allocate rather than a container to fill: deduplication, ordering, truncation policy, and source identifiers for citation.
What to ask your recruiter
Before you prepare anything, send one short email: how many rounds, what each one is called, how long each runs, whether any of them is a take-home, and whether an AI assistant is allowed in the coding round. Recruiters answer this routinely. Anything they confirm outranks every pattern on this page, because it is first-party evidence about your loop rather than a general tendency across companies.
What it is not
- Not a transformer architecture quiz. You will rarely be asked to derive attention.
- Not a prompt-writing exercise. Prompt text is a detail inside the design, not the design.
- Not a trivia round about the newest model release.
- Not an excuse to skip fundamentals. Storage, queues and idempotency are still graded.
How to spend forty-five minutes
The most common way to lose this round is to run out of time before the sections that are graded hardest.
| Minutes | Section | Why it matters |
|---|---|---|
| 0–6 | Requirements, definition of correct, constraints | Rounds are frequently decided here. Designing before defining correct is the classic error. |
| 6–16 | Data path and retrieval | Where depth is expected. Take a position on chunking and hybrid search. |
| 16–26 | Serving path, context construction, caching | Token budget, streaming, and what is deterministic and stays that way. |
| 26–35 | Evaluation loop | Eval set, metric, gate, online signal. Commonly the highest-signal section. |
| 35–45 | Failure handling, cost, scale | Unavailable, slow, confidently wrong, bad context — and what the user sees in each. |
Study now, review, skip
- One AI feature you shipped, drawable end to end in under five minutes.
- A concrete eval story: baseline, eval set, metric, failure found, change, measurable result.
- Your real numbers for one system: p95 latency, cost per request, and what dominates each.
- Two production failure modes you personally debugged, and how you detected them.
- Hybrid search and reranking trade-offs.
- Caching with non-deterministic output.
- Streaming and perceived latency versus total latency.
- Attention and architecture derivations.
- Recent model release comparisons.
- Prompt technique lists.
Common failure modes
- Designing before defining what a correct output is.
- A detailed ingestion pipeline and no time left for evaluation or reliability.
- "We embed the documents and search" with no position on chunking or lexical search.
- No fallback path, so the design has nothing to say during a provider outage.
- Treating a confidently wrong answer as the same failure as a timeout.
- Inventing numbers that do not survive one arithmetic follow-up.
Common questions
- Is an LLM system design interview different from a normal system design interview?
- It is the same round with four additions: a retrieval path, an evaluation loop, token-based cost, and a failure class where the component returns a confident wrong answer. The fundamentals still carry most of the weight.
- Do I need to know vector database internals?
- Usually only enough to reason about index type, recall against latency, filtering and freshness. Deep internals are rarely the signal outside infrastructure roles.
- How much detail should I give on prompts?
- Enough to show the structure and the instruction the design depends on — grounding, refusal, citation. Writing prompt text on the whiteboard is almost always a poor use of the time.
- What if the interviewer keeps pushing on failure cases?
- That is the round working as intended. Name the failure class, say how you detect it, say what the system does, and say what the user sees. Those four beats answer most follow-ups.
Related interview guides
- AI system design interview: framework and prep
A reusable framework for designing systems where a model is one dependency among many.
- RAG interview questions: retrieval, evals and failure modes
Ingestion through reranking to context construction, centred on production trade-offs.
- LLM evaluation interview: evals, judges and regressions
Golden sets, task metrics, judge models, online monitoring, and agent evaluation.
- LLM engineer interview: rounds and prep
Serving, retrieval, agents, evaluation, and the latency and cost judgment this title is screened on.
How every ranking on this site is reasoned about, including what counts as evidence: read the methodology.
Map your actual interview.
This page is the general pattern. Paste the job you are actually interviewing for and the wording your recruiter used, and the map ranks the rounds you are likely to face, explains why each is ranked where it is, and tells you what to skip.
See a complete Full Interview Map example