RAG interview questions
RAG interview questions test whether you can make retrieval work under production conditions, not whether you can define retrieval-augmented generation. Interviewers move quickly to chunking strategy, hybrid search, reranking, what happens when the top results are irrelevant, how retrieval quality is measured separately from answer quality, how the index stays fresh, and how per-user access control survives an embedding index. Trade-offs are the answer; pipeline diagrams are not.
Applies to any round where documents, knowledge bases or grounded answering come up — which in practice is most LLM and applied AI system design rounds, whether or not the word RAG is used.
Why retrieval is where design rounds are won and lost
Almost every candidate can draw the pipeline: chunk the documents, embed them, store the vectors, embed the query, take the top results, put them in the prompt. Because everyone can draw it, drawing it earns nothing. The round is decided by what you say about the parts that are genuinely hard.
Those parts are consistent. Chunking destroys or preserves meaning depending on the document. Pure vector search misses exact identifiers, product codes and rare terms. The top result is often not the best one. Context is a budget, not a container. Documents change, and stale answers are worse than no answers. And permissions do not disappear because the text is now a vector.
Interviewers tend to probe the second and third of those first, because a candidate who has only read about RAG usually has no position on either.
The pipeline, and the question asked at each stage
For each stage: what interviewers probe, and the answer that indicates production experience.
- Ingestion and parsing
- Probed: how you handle PDFs, tables, code, and documents with meaningful structure. Strong answers mention that parsing quality caps everything downstream, and that tables and headers usually need dedicated handling rather than being flattened into prose.
- Chunking
- Probed: your strategy and why. Strong answers reject a single global chunk size, use document structure where it exists, keep headings or titles attached to chunks for context, and note that overlap trades index size against boundary loss.
- Embeddings
- Probed: model choice, dimensionality, domain mismatch, and multilingual behaviour. Strong answers raise the migration problem: changing the embedding model means reindexing everything, so the choice has an operational cost.
- Indexing and filtering
- Probed: index type, recall against latency, and metadata filtering. Strong answers note that filters applied after retrieval can empty a result set, so filters generally belong in the query.
- Retrieval and hybrid search
- Probed: whether you use lexical search alongside vectors. Strong answers explain that vector search alone fails on exact identifiers, error codes and rare proper nouns, and describe how results from both are combined.
- Reranking
- Probed: whether you retrieve wide and rerank narrow. Strong answers state the cost: a cross-encoder reranker adds latency and spend, and is justified by measured precision at the top of the list rather than by default.
- Context construction
- Probed: what actually goes into the prompt. Strong answers treat context as a budget with an allocation, deduplicate near-identical chunks, order deliberately, and include source identifiers so answers can cite.
- Generation and grounding
- Probed: what the model is instructed to do when the context does not contain the answer. Strong answers include an explicit refusal path and citation of retrieved sources.
What to ask your recruiter
Before you prepare anything, send one short email: how many rounds, what each one is called, how long each runs, whether any of them is a take-home, and whether an AI assistant is allowed in the coding round. Recruiters answer this routinely. Anything they confirm outranks every pattern on this page, because it is first-party evidence about your loop rather than a general tendency across companies.
Diagnosing a bad answer: which stage failed
Being able to run this triage out loud is one of the strongest signals available in this round.
| Symptom | Likely stage | How you would confirm |
|---|---|---|
| Answer is wrong and cites nothing relevant | Retrieval | Inspect the retrieved chunks for the query. If the correct passage is absent, the generation step was never given a chance. |
| Correct passage retrieved, answer still wrong | Context construction or generation | Check position and truncation in the assembled prompt, then test the same context with the instruction tightened. |
| Works for common queries, fails on product codes | Lexical gap | Compare vector-only against hybrid results on a set of identifier queries. |
| Answer is right but out of date | Freshness | Check the index timestamp for the source document against its last modification. |
| User sees content they should not | Access control | Verify permissions are enforced as a pre-filter in the query, not applied to results after retrieval. |
| Quality dropped after a deploy with no model change | Ingestion or chunking regression | Diff index statistics: chunk counts, average length, parse failures. |
The questions that separate candidates
Themes, not a bank. Prepare a position on each, with the trade-off attached.
- How do you measure retrieval quality separately from answer quality?
- The expected answer distinguishes the two: a labelled set of queries with known relevant documents, measured with recall at k and precision at the top of the list, evaluated independently of whatever the model then writes.
- What do you do when the top results are irrelevant?
- Detect it, then decline. A low-confidence path that says the answer is not in the documents is usually better product behaviour than a fluent guess, and it needs a threshold you can defend.
- How does per-user access control work over an embedding index?
- Permissions as indexed metadata and enforced as a pre-filter, re-checked at read time against the source of truth, with deletion and revocation propagating to the index.
- How do you keep the index fresh?
- Change-driven reindexing rather than full rebuilds, deletions honoured promptly, and a monitored lag metric between source change and index update.
- Retrieval is adding 400ms. What do you cut?
- Measure the split between embedding, search, and reranking first, then decide: narrow the rerank set, cache query embeddings, or reduce candidate count — with a measured quality check after.
- Should this be RAG at all?
- Sometimes the corpus is small enough to include wholesale, or structured enough that a query beats a search. Volunteering that is a strong signal of judgment.
- How do you handle prompt injection in retrieved content?
- Treat retrieved text as untrusted input: keep it out of the instruction channel, constrain tool access downstream, and never let document content escalate the model's permissions.
Study now, review, skip
- A defensible chunking position, including when you would change it.
- Why hybrid search exists, and the query classes that break vector-only retrieval.
- Retrieval evaluation as distinct from answer evaluation.
- The failure triage table above, until you can run it out loud.
- Reranking cost and where it pays for itself.
- Access control and deletion propagation.
- Freshness and reindexing strategy.
- Vector database vendor feature matrices.
- Embedding benchmark leaderboards.
- Framework tutorials; nobody is grading which library you used.
Common failure modes
- Describing the pipeline and stopping, as though the diagram were the answer.
- One global chunk size with no reasoning about document structure.
- No lexical search, and no awareness of what vector-only retrieval misses.
- Measuring only end-to-end answer quality, so a retrieval regression is invisible.
- Applying permission filters after retrieval, which both leaks and silently empties results.
- Filling the context window because it is large, rather than allocating it deliberately.
- Treating retrieved document text as trusted instructions.
A three-day plan
Day 1 — Take positions
- Write your chunking, hybrid search and reranking positions in one paragraph each, with the trade-off named.
- Write down the query classes where your current or last system fails.
Day 2 — Measurement
- Design a retrieval eval on paper: fifty queries, known relevant documents, recall at k, precision at the top.
- Decide what threshold would trigger the no-good-context refusal path.
Day 3 — Triage out loud
- Run the failure triage table as a spoken exercise against three invented symptoms.
- Prepare the access control and freshness answers, which are the two most commonly skipped.
Common questions
- Is RAG still asked about in 2026 interviews?
- The acronym appears less often than it used to, but the content is constant: grounding answers in a corpus you control remains the core of most AI product systems, so the questions show up inside system design rounds whether or not the term is used.
- Do long context windows make retrieval unnecessary?
- They change the trade-off rather than removing it. Larger context costs more per request, adds latency, and does not help when the corpus is far larger than any window. Saying that clearly is usually treated as a strong answer.
- How deep should I go on embedding models?
- Deep enough to justify a choice on domain fit, language coverage, dimensionality and the reindexing cost of changing later. Reciting benchmark rankings is rarely what the question is after.
- What if I have only used a managed retrieval service?
- That is fine if you can say what it does on your behalf and where its defaults would break for your data. The weak version of that answer is not knowing what the service decided for you.
Related interview guides
- LLM system design interview, decoded
The LLM-specific cut: retrieval architecture, prompting and orchestration, token budgets, failure handling.
- LLM evaluation interview: evals, judges and regressions
Golden sets, task metrics, judge models, online monitoring, and agent evaluation.
- LLM interview questions, organised by what they test
Nine themes instead of a question bank, and why memorised answers fall apart under follow-up.
- AI system design interview: framework and prep
A reusable framework for designing systems where a model is one dependency among many.
How every ranking on this site is reasoned about, including what counts as evidence: read the methodology.
Map your actual interview.
This page is the general pattern. Paste the job you are actually interviewing for and the wording your recruiter used, and the map ranks the rounds you are likely to face, explains why each is ranked where it is, and tells you what to skip.
See a complete Full Interview Map example