The LLM engineer interview
An LLM engineer interview sits closer to the model than a general AI engineer loop. Expect an LLM system design round covering serving, retrieval and context construction, an evaluation conversation, a practical coding session, and a project deep dive. The distinguishing content is depth on inference economics — token budgets, caching, batching, model and provider selection — and on measuring quality when there is no single correct output.
This covers LLM, GenAI and foundation-model product engineering roles. Inference-kernel, training-infrastructure and research roles are a different loop and are not covered here.
How this differs from a generic AI engineer loop
Both loops test production judgment. The LLM engineer loop adds depth in three places: how generation is served, how context is assembled, and how quality is measured when the output is free text.
In practice that means the design round goes further down. Instead of stopping at "we call the model", interviewers ask what the token budget is, what is cached and at what granularity, whether streaming changes the user-visible latency story, what happens when the context window fills, and how you would decide between a large hosted model and a smaller one you run yourself.
The other shift is provider reality. Rate limits, quota, version changes, deprecations and regional availability are operational facts in this role, and candidates who have lived with them tend to mention them unprompted. That is itself a signal.
What to expect: the usual loop
| Round | Typical length | What it tests |
|---|---|---|
| Recruiter screen | 25 min | Level, scope of the role, and the round names. Ask whether evaluation is a separate round. |
| LLM system design | 60 min | Serving, retrieval, context construction, caching, fallbacks, and the cost and latency envelope. |
| Evaluation & reliability | 45 min | Golden sets, task metrics, judge models, regression gates, and online monitoring. |
| Practical coding | 45–60 min | Reading and changing real code. Often an integration or data-handling task rather than an algorithm puzzle. |
| Project deep dive | 45–60 min | One generation feature you shipped, in production detail: what broke, what you measured, what changed. |
| Behavioral | 45 min | Judgment under ambiguity, and how you handled shipping something you could not fully verify. |
What to ask your recruiter
Before you prepare anything, send one short email: how many rounds, what each one is called, how long each runs, whether any of them is a take-home, and whether an AI assistant is allowed in the coding round. Recruiters answer this routinely. Anything they confirm outranks every pattern on this page, because it is first-party evidence about your loop rather than a general tendency across companies.
What interviewers are testing
- Serving and inference economics
- Token accounting, prompt and prefix caching, batching, streaming, context window management, and where each of those actually moves p95 latency or unit cost.
- Retrieval architecture
- Chunking, indexing, hybrid search, reranking, and freshness — plus what you do when the top results are useless. Vague retrieval answers are among the most common weaknesses in this round.
- Evaluation design
- A golden set with known answers, a task-appropriate metric, a judge model used carefully rather than trusted blindly, and a regression gate that blocks a release.
- Model and provider selection
- Being able to justify a choice on quality, latency, cost, privacy and operational risk together, instead of naming the model with the best benchmark.
- Failure handling
- Timeouts, retries that do not amplify load, partial output, refusal handling, and what the user sees when generation fails or is unusable.
- Data boundaries
- What leaves your infrastructure, what is retained by a provider, what is logged, and which categories of input must never reach either.
Question themes that recur
Patterns, not a question bank. The follow-ups matter more than the opener.
- Design a support assistant over our documentation
- Retrieval, grounding, refusal behaviour, escalation to a human, and how you would measure answer correctness without a reference answer.
- Your p95 latency doubled overnight. Diagnose it
- Tests instrumentation. Strong answers separate retrieval, prompt assembly, generation, and network, and know which they can observe today.
- Cut the cost of this feature in half without users noticing
- Caching, shorter context, cheaper model for a subset of traffic, routing by difficulty, and how you would verify quality held.
- How do you stop a prompt change from regressing quality?
- Versioning, an eval suite in CI, a held-out set, and a gate. "We review the diff" is not an answer.
- When would you fine-tune rather than improve retrieval or prompting?
- Tests whether you understand that most quality problems in production are context problems, and what the ongoing cost of a tuned model is.
- The model is confidently wrong on a class of inputs. What now?
- Detection first, then containment: guardrails, refusal, routing, and an eval case that keeps the fix from regressing.
Study now, review, skip
- Your own numbers: tokens per request, p95, cost per thousand requests, cache hit rate.
- One retrieval design you can defend, including its known weaknesses.
- An evaluation story: golden set, metric, a regression it caught.
- A concrete cost or latency reduction you shipped, with the before and after.
- Reranking and hybrid search trade-offs.
- Structured output, tool calling, and schema validation failure handling.
- Guardrail approaches and their false-positive costs.
- Attention derivations and architecture internals, unless the role is inference or training.
- Comparing this month's model releases; the round is about method, not the leaderboard.
- Prompt-engineering trick lists.
Common failure modes
- Describing retrieval as "we embed the docs and search" with no position on chunking, hybrid search, or what happens on a bad top-k.
- Optimising cost by guessing instead of measuring where tokens are actually spent.
- Using a judge model as ground truth with no human-labelled set behind it.
- No versioning story for prompts, so no way to attribute a quality change to a cause.
- Ignoring the failure path entirely: no timeout, no partial output, no user-visible degraded state.
- Treating the context window as free and filling it, rather than treating context as a budget to allocate.
A three-day plan
Day 1 — Numbers and one system
- Reconstruct real numbers for one generation feature you worked on.
- Draw it end to end, including the eval loop, in five minutes.
- Write down its two worst failure modes and how they were detected.
Day 2 — Retrieval and evaluation depth
- Take a position on chunking and hybrid search you can defend, with the trade-off stated.
- Design an eval suite for that feature on paper: set, metric, judge role, gate.
- Prepare the answer to "how would you halve the cost?" with a measurement step first.
Day 3 — Coding and delivery
- 45 minutes in an unfamiliar repo: change behaviour, add a test, narrate the debug loop.
- Rehearse the design round out loud, finishing at failure handling rather than architecture.
- Prepare questions about how the team evaluates changes today.
Common questions
- Is an LLM engineer interview harder than an AI engineer interview?
- Not harder, narrower. The same production judgment is expected, but the design and evaluation rounds usually go deeper into serving, retrieval and measurement, and shallow answers there are more visible.
- Do I need to have trained or fine-tuned a model?
- For most LLM engineer product roles, no. What matters is whether you can reason about when tuning is the right lever versus retrieval, context, or routing — and what it costs to maintain afterwards.
- How much does the specific provider or framework matter?
- Less than candidates expect. Interviewers generally care that you can justify a choice on quality, latency, cost, privacy and operational risk. Naming a preferred stack without those reasons is a weak answer.
- Will there be a take-home?
- It varies by company and is genuinely unknown until the recruiter says. If there is one, the review conversation afterwards usually carries more weight than the code itself.
Related interview guides
- LLM system design interview, decoded
The LLM-specific cut: retrieval architecture, prompting and orchestration, token budgets, failure handling.
- RAG interview questions: retrieval, evals and failure modes
Ingestion through reranking to context construction, centred on production trade-offs.
- LLM evaluation interview: evals, judges and regressions
Golden sets, task metrics, judge models, online monitoring, and agent evaluation.
- AI engineer interview: rounds and prep
The generalist AI engineering loop: coding, AI system design, project deep dive, behavioral.
How every ranking on this site is reasoned about, including what counts as evidence: read the methodology.
Map your actual interview.
This page is the general pattern. Paste the job you are actually interviewing for and the wording your recruiter used, and the map ranks the rounds you are likely to face, explains why each is ranked where it is, and tells you what to skip.
See a complete Full Interview Map example