The AI system design interview
An AI system design interview asks you to design a product system in which one or more models are dependencies with variable output quality and token-based cost. It is a standard system design round plus four additions: a data and retrieval path, an evaluation loop, a cost and latency budget, and explicit failure handling for when the model is wrong or unavailable. The fundamentals still decide most of the outcome.
This is the broad version, covering classical ML, LLM and hybrid systems. For the language-model-specific cut, read the LLM system design guide. For product-scoping-shaped rounds, read the AI design round guide.
What changes when a model is in the diagram
Everything you already know about system design still applies. Interfaces, data flow, storage, queues, caching, idempotency, rate limits, observability and failure isolation are what the round is mostly graded on, and candidates who abandon them to talk about models tend to score badly.
Four things are genuinely new. First, one dependency returns output of variable quality rather than simply succeeding or failing, so correctness becomes a distribution rather than a boolean. Second, that dependency is often priced per token, which makes payload size an architectural concern rather than a detail. Third, quality cannot be asserted in a unit test, so the system needs an evaluation loop as a first-class component. Fourth, the model can be confidently wrong, which is a different failure class from being down and needs different handling.
A good answer treats the model as one component in a system you control, rather than treating the system as scaffolding around a model.
A framework you can run in any AI design round
Seven steps, roughly forty-five minutes. Say the step names out loud as you go; interviewers grade structure as well as content.
- Requirements and scope. Who the user is, what a correct output looks like, the volume, and the latency the experience can tolerate. Write the definition of correct down before designing.
- Non-goals and constraints. What you are deliberately not building, plus the data, privacy and regulatory boundaries. Name them early; they eliminate whole designs.
- Data and retrieval path. Where the knowledge comes from, how it is ingested and kept fresh, how it is indexed and selected, and what is sent to the model.
- Core design and model placement. The request path end to end, what the model is responsible for, and which parts stay deterministic because they can be.
- Evaluation loop. The eval set, the metric, the gate that blocks a release, and the online signal that tells you quality moved after deploy.
- Reliability and failure handling. Timeouts, retries that do not amplify load, fallbacks, degraded modes, and what the user sees for each failure class.
- Cost, latency and scale. Where the time and the money go, what you would cut first under a hard budget, and how the design behaves at ten times the traffic.
What to ask your recruiter
Before you prepare anything, send one short email: how many rounds, what each one is called, how long each runs, whether any of them is a take-home, and whether an AI assistant is allowed in the coding round. Recruiters answer this routinely. Anything they confirm outranks every pattern on this page, because it is first-party evidence about your loop rather than a general tendency across companies.
Four failure classes, and what each one needs
Naming these separately is one of the fastest ways to sound like someone who has run this in production.
| Failure class | What it looks like | What the design needs |
|---|---|---|
| Unavailable | Provider outage, rate limit, timeout. | Timeouts, bounded retries with backoff, a cached or deterministic fallback, and a clear user-visible degraded state. |
| Slow | p95 blows past the experience budget under load. | Streaming, smaller context, cheaper model for a traffic slice, caching, and a hard deadline with a partial result. |
| Wrong but confident | A fluent, plausible, incorrect answer. | Grounding and citation, validation against a source of truth, guardrails, refusal paths, and an eval case that catches the class. |
| Wrong because the input was wrong | Retrieval returned nothing useful, or the data is stale. | Retrieval quality metrics, a no-good-context path that declines to answer, and freshness monitoring on the index. |
What interviewers are testing
- Requirement discipline
- Whether you define correct, fast enough, and affordable before drawing boxes. Rounds are frequently decided in the first five minutes.
- Model placement judgment
- Keeping deterministic work deterministic, and being able to say which part of the system genuinely needs a model.
- Evaluation as architecture
- Treating the eval set, the gate and the online metric as components with owners and storage, not as an afterthought.
- Numbers
- Rough but honest estimates: requests per second, payload size, cost per thousand requests, and where p95 is spent.
- Degradation design
- What the product does — visibly, to the user — in each of the four failure classes above.
- Data boundaries
- What is logged, what is retained, what crosses into a third-party provider, and what is never allowed to.
Study now, review, skip
- The seven-step framework, until you can run it without notes.
- One design you can draw end to end in five minutes, including the eval loop.
- The four failure classes and the handling each one needs.
- Rough cost and latency arithmetic for a model-backed request.
- Retrieval design: chunking, hybrid search, reranking, freshness.
- Caching with non-deterministic output, and where it is unsafe.
- Standard system design: queues, idempotency, sharding, backpressure.
- Architecture internals of the models themselves.
- Vendor feature comparisons.
- Memorised reference architectures. Interviewers push on the parts you did not reason about.
Common failure modes in this round
- Drawing the architecture before defining what a correct output is.
- A beautiful happy path and nothing to say when retrieval returns junk.
- Evaluation answered with a public benchmark name instead of a task-specific eval set.
- No numbers at all, or numbers so confident they invite a follow-up the candidate cannot answer.
- Putting the model in the path of work that is a database query.
- Running out of time in ingestion detail and never reaching reliability or cost, which are usually the graded sections.
A three-day plan
Day 1 — Framework fluency
- Memorise the seven steps and run one prompt end to end out loud, timed at forty-five minutes.
- Record yourself. Most candidates discover they spent twenty minutes on ingestion.
Day 2 — The graded sections
- Run two more prompts but start at evaluation and reliability, to force fluency where time usually runs out.
- Write your default answers for each of the four failure classes.
Day 3 — Numbers and trade-offs
- Practise rough estimates: payload size, requests per second, cost per thousand requests, p95 breakdown.
- For one design, name what you would cut first under a hard latency budget and under a hard cost budget, and why they differ.
Common questions
- How is this different from a normal system design interview?
- It is a normal system design interview with four additions: a data and retrieval path, an evaluation loop, token-based cost, and a failure class where the component returns a confident wrong answer. The distributed systems content still carries most of the weight.
- Do I need to know vector database internals?
- Usually only enough to reason about trade-offs: index type, recall against latency, filtering, and freshness. Deep internals are rarely the signal unless the role is specifically infrastructure.
- Should I propose fine-tuning?
- Only with a reason. In most product designs, retrieval quality, context construction and routing move quality further and faster, and a tuned model adds an ongoing maintenance and evaluation cost you should acknowledge.
- What if I run out of time?
- Say what you would cover next and give the one-sentence version of each. Reaching evaluation and reliability shallowly is better than never reaching them, because those sections are frequently where the round is graded.
Related interview guides
- LLM system design interview, decoded
The LLM-specific cut: retrieval architecture, prompting and orchestration, token budgets, failure handling.
- The AI design round, decoded
The most overloaded round name in the loop, and how to work out which of the three versions you are getting.
- RAG interview questions: retrieval, evals and failure modes
Ingestion through reranking to context construction, centred on production trade-offs.
- LLM evaluation interview: evals, judges and regressions
Golden sets, task metrics, judge models, online monitoring, and agent evaluation.
How every ranking on this site is reasoned about, including what counts as evidence: read the methodology.
Map your actual interview.
This page is the general pattern. Paste the job you are actually interviewing for and the wording your recruiter used, and the map ranks the rounds you are likely to face, explains why each is ranked where it is, and tells you what to skip.
See a complete Full Interview Map example