The LLM evaluation interview
An LLM evaluation interview asks how you know a probabilistic system works before users tell you. Expect questions about building a golden set, choosing a task-appropriate metric, using judge models without trusting them, keeping humans in the loop affordably, gating releases on regressions, and monitoring quality online after deploy. Evaluation is the most common weak spot in AI engineering loops, which makes it the highest-return thing to prepare.
Covers dedicated evaluation rounds and the evaluation portion of system design and deep-dive rounds, which is where most of these questions actually arrive.
Why evaluation carries so much weight
Every other part of an AI system can be reasoned about from general engineering experience. Evaluation cannot. It is the part that only appears once something has shipped and someone has had to answer whether a change made things better or worse, with money and reputation attached to the answer.
That makes it an efficient filter. A candidate who has genuinely run an evaluation loop talks about label disagreement, the cost of human review, the set they had to rebuild because it did not reflect real traffic, and the regression they only caught because a gate existed. A candidate who has not tends to name a benchmark and stop.
It is also the highest-return preparation available, because the bar is low in practice. One concrete, honest evaluation story with a baseline, a metric, a discovered failure and a measured result will outperform most of the field.
The evaluation ladder
Interviewers frequently want to hear these as distinct layers with different costs and different purposes. Stating them in order is a strong opening.
- Assertions and unit checks. Cheap, deterministic, and run on every commit: schema validity, required fields, forbidden content, length and format. They catch breakage rather than quality.
- A golden set. A fixed collection of inputs with known-good outputs or graded criteria, drawn from real traffic rather than invented, large enough to be stable and small enough to afford.
- Task metrics. Chosen for the task: exact match or accuracy where there is a correct answer, retrieval metrics where grounding matters, rubric scores where there is not a single right output.
- Judge models. An LLM scoring outputs against a rubric, calibrated against human labels on a subset, with its own agreement rate tracked as a number.
- Human review. Expensive and irreplaceable. Used on a sample, on disagreements, and on anything the judge scores near a threshold.
- Shadow and online. Running the new version against real traffic without showing it, then a small online test with a guardrail metric that can stop the rollout.
- Production monitoring. Ongoing quality signals, user feedback, refusal and error rates, and drift detection on inputs as well as outputs.
What to ask your recruiter
Before you prepare anything, send one short email: how many rounds, what each one is called, how long each runs, whether any of them is a take-home, and whether an AI assistant is allowed in the coding round. Recruiters answer this routinely. Anything they confirm outranks every pattern on this page, because it is first-party evidence about your loop rather than a general tendency across companies.
The questions that separate candidates
- How would you build a golden set for this feature?
- Expected: sampled from real traffic including the failures, stratified across the input classes you care about, labelled with a written rubric, with disagreement between labellers measured rather than assumed away.
- How large does the set need to be?
- Expected: enough that the metric does not move on noise, which depends on effect size rather than on a round number. Strong answers mention running the same version twice to see the variance before trusting a difference.
- Can you trust an LLM judge?
- Expected: only as far as it agrees with humans on a labelled subset, with that agreement measured and rechecked. Known biases — position, length, self-preference — should be named and controlled for.
- Offline says better, online says worse. What happened?
- Expected: distribution mismatch between the eval set and real traffic, a metric that does not track user value, or an interaction effect such as latency degrading the experience the quality gain was supposed to improve.
- How do you stop a prompt change regressing quality?
- Expected: versioned prompts, the eval suite running in CI, a held-out set that is not used during iteration, and a gate that blocks the release rather than a report nobody reads.
- How do you evaluate an agent?
- Expected: task-level success on a fixed scenario set, sampled trajectory review, partial credit for multi-step tasks, and separate accounting for cost and step count, which can regress while success stays flat.
- How do you evaluate safety or refusal behaviour?
- Expected: a dedicated adversarial set, both directions measured — harmful output rate and false refusal rate — because tightening one usually worsens the other.
- What do you monitor in production?
- Expected: a sampled quality signal rather than only system metrics, user feedback and its known bias toward complaints, refusal and error rates, latency and cost per request, and drift in the input distribution.
Choosing a metric for the task
Interviewers often probe whether you match the metric to the task shape.
| Task shape | Reasonable primary metric | What it misses |
|---|---|---|
| Extraction or classification | Accuracy, or precision and recall per class. | Says nothing about formatting, tone, or the cost of a confident error. |
| Grounded question answering | Correctness against a reference plus citation validity. | Misses retrieval failures unless retrieval is measured separately. |
| Summarisation or free text | Rubric score from a calibrated judge, sampled to humans. | Judge bias, and drift in the rubric's interpretation over time. |
| Code generation | Tests passing on a held-out task set. | Readability, security, and whether the change was the right one. |
| Multi-step agent task | Task success on a scenario set, with step and cost budgets. | Near-misses and unsafe paths that still reached a correct end state. |
Study now, review, skip
- One real evaluation story: baseline, set, metric, failure found, change, result.
- The evaluation ladder, in order, with the cost of each layer.
- Judge calibration: how you would check agreement and what you would do if it was poor.
- The offline-versus-online divergence answer.
- Retrieval metrics as a separate measurement from answer quality.
- Agent trajectory evaluation and partial credit.
- Safety evaluation in both directions, including false refusals.
- Public benchmark leaderboards. They almost never answer the question being asked.
- Metric formulas in the abstract, without a task attached.
- Evaluation tooling comparisons.
Common failure modes
- Naming a public benchmark when asked how you would evaluate a specific product feature.
- Treating a judge model as ground truth, with no human-labelled subset behind it.
- An eval set built from invented examples, so it never contains the failures real traffic produces.
- Iterating against the same set until the score is meaningless, with no held-out data.
- Reporting a single aggregate number with no variance, so noise reads as improvement.
- No gate: the evaluation exists, produces a report, and blocks nothing.
- Forgetting cost and latency are regressions too, even when quality is flat.
A three-day plan
Day 1 — Your story
- Write one evaluation you ran: what you baselined, what the set was, what the metric was, what it caught.
- Include the part that went wrong. Rebuilt sets and bad labels are credibility, not weakness.
Day 2 — Design one from scratch
- Take a feature you know and design its evaluation on paper, through all seven ladder layers.
- Decide the gate: what number blocks a release, and who can override it.
Day 3 — Judges and online
- Write your judge calibration procedure and the biases you would control for.
- Prepare the offline-versus-online divergence answer and the guardrail metric you would ship with.
Common questions
- Is there usually a separate evaluation round?
- Sometimes, and it is becoming more common in LLM-heavy roles. Often it is folded into system design or the project deep dive instead. Your recruiter can confirm whether it is a named round in your loop.
- What if my team never had a formal evaluation process?
- Say so and describe what you did instead — spot checks, user complaints, a rollback after a bad release — then describe what you would build now and why. An honest account of an immature process is far stronger than an invented mature one.
- Are public benchmarks ever the right answer?
- As a coarse model-selection filter before you build your own set, occasionally. As an answer to whether your feature works for your users, essentially never, and saying that clearly is itself a positive signal.
- How much statistics do I need?
- Enough to know that a small difference on a small set is usually noise, and to say how you would check. Formal significance testing is rarely required, but the instinct to ask whether a change is real is.
Related interview guides
- RAG interview questions: retrieval, evals and failure modes
Ingestion through reranking to context construction, centred on production trade-offs.
- Agentic AI interview: rounds and prep
State, tools, orchestration, recovery, guardrails, and how agent reliability is actually probed.
- LLM system design interview, decoded
The LLM-specific cut: retrieval architecture, prompting and orchestration, token budgets, failure handling.
- LLM interview questions, organised by what they test
Nine themes instead of a question bank, and why memorised answers fall apart under follow-up.
How every ranking on this site is reasoned about, including what counts as evidence: read the methodology.
Map your actual interview.
This page is the general pattern. Paste the job you are actually interviewing for and the wording your recruiter used, and the map ranks the rounds you are likely to face, explains why each is ranked where it is, and tells you what to skip.
See a complete Full Interview Map example