RAG interview questions
What interviews ask about retrieval-augmented generation: chunking and ingestion, embeddings and vector search, hybrid search and reranking, and how you measure whether retrieval and answers are any good.
25 questions across 7 topics. Answer each out loud, then open the points a strong answer covers.
Retrieval for RAG: chunking, hybrid search, reranking
Asked in most interview loops
Retrieval keeps missing answers that are clearly in the documents; what would you try first?
- Check chunking: chunks cut mid-thought or stripped of their document title lose context, so use structure-aware splits, overlap or prepended context.
- Add BM25 beside embeddings, since keywords catch exact names and codes while embeddings catch paraphrases, and fuse the lists with reciprocal rank fusion.
- Measure every change with recall@k on a labelled or synthetic question set, so you know retrieval improved rather than just changed.
Where does a reranker sit in a RAG pipeline, and what does it cost you?
- First-stage retrieval cheaply pulls a wide candidate set with BM25 and vector search, favouring recall over precision.
- A cross-encoder reranker reads the query and each candidate together, scoring relevance more accurately than comparing separate embeddings.
- It adds latency and cost per candidate, so you cap the candidate count and pass only the top few chunks to the LLM.
How would you choose a chunk size, and what goes wrong at each extreme?
- Small chunks give focused embeddings and little noise, but they can split an answer across chunks or lose context such as which product a sentence is about.
- Large chunks keep context but mix several topics into one embedding, which weakens its match to a specific query and spends more of the prompt on irrelevant text.
- Sweep a few sizes and overlaps on your eval set, measuring recall@k and end-to-end answer quality, since the best size depends on your documents and queries.
How does reciprocal rank fusion merge BM25 and vector results, and why not just add the scores?
- RRF gives each document the sum over result lists of 1/(k + rank), with k commonly 60, so documents ranked well by both retrievers rise to the top.
- It uses ranks only, so you never have to put unbounded, query-dependent BM25 scores and cosine similarities onto a common scale before combining them.
- The cost is discarding score magnitudes, so a tuned, normalised weighted fusion can sometimes beat it; compare both on recall@k or NDCG before choosing.
Users ask about the current version of a policy, but retrieval returns chunks from every old version too; how would you fix it?
- Embeddings can’t tell versions apart, because old and new chunks say almost the same thing, so similarity alone has no reason to prefer the current one.
- Attach version, effective date and source as metadata at ingestion, then filter or boost on them, extracting the filter from the question with rules or an LLM.
- Label a set of version- and date-specific questions and measure how often the top-k chunks come from the correct version, before and after the change.
Embeddings and vector search
Asked in most interview loops
When would you use cosine similarity versus dot product for vector search?
- Cosine compares direction only and ignores vector length; dot product also rewards magnitude, which some models are trained to use.
- On L2-normalised vectors cosine and dot product give identical rankings, so you can normalise once and use the cheaper dot product.
- Use the metric the embedding model was trained with; mismatching it quietly hurts retrieval quality without raising any error.
How does HNSW find nearest neighbours fast, and which knobs trade recall for latency?
- HNSW builds a layered graph: sparse upper layers for long hops, dense lower layers for fine search, walked greedily from the top.
- M sets links per node, trading memory and build time for recall; ef_construction sets how carefully the graph is built.
- ef_search sets the candidate list size at query time: higher gives better recall but slower queries, tuned against exact search results.
What is the difference between symmetric and asymmetric semantic search, and why does it change your model choice?
- Symmetric search compares texts of similar form, like finding duplicate questions; asymmetric search matches a short query to a longer passage that answers it.
- Models trained for asymmetric retrieval learn to place a question near its answer even though the two look different, while a symmetric model may surface similar questions instead.
- Many asymmetric models expect different prefixes or instructions for queries and documents, and leaving them out quietly lowers recall, so follow the model card and check on your eval set.
When is exact nearest-neighbour search good enough, and what do you give up by moving to an approximate index?
- Flat search compares the query with every vector, so it always returns the true neighbours, but its cost grows linearly with corpus size and dimension.
- Approximate indexes such as HNSW or IVF visit only a fraction of the vectors, trading a little recall and extra memory or build time for much lower latency.
- Run sample queries through both, measure the approximate index’s recall@k against the exact results, and pick the cheapest settings that meet your recall and latency targets.
You want to switch to a newer embedding model; what does the migration involve?
- Vectors from different models live in unrelated spaces, so the whole corpus must be re-embedded; mixing old and new vectors in one index makes similarities meaningless.
- Dimension, similarity metric, maximum input length and query prefixes may all change, which affects index configuration, chunk size and storage cost.
- Compare both models on your own labelled retrieval set with recall@k or NDCG, since leaderboard rankings may not hold on your domain, then build a new index and switch over.
Evaluating RAG: retrieval metrics and faithfulness
Asked in most interview loops
How would you evaluate retrieval and generation separately in a RAG system?
- Score retrieval with recall@k, MRR or NDCG against questions labelled with the chunks that answer them; context precision also rewards good rank order.
- Score generation for faithfulness, supported claims divided by total claims, and answer relevance, usually with an LLM judge or NLI model.
- With no labels, generate synthetic questions from your own chunks, then hand-check a sample so the set resembles real user questions.
Faithfulness went up after your change; does that mean the product got better?
- Not necessarily: faithfulness only says claims are backed by the context, so shorter or more hedged answers can score higher.
- Check context recall, which needs a reference answer, and answer relevance too, since a faithful answer built on the wrong chunks still fails users.
- The LLM judge inside these metrics can be wrong, so spot-check its claim splits and verdicts against human labels before trusting the trend.
When would you report recall@k, MRR or NDCG for a RAG retriever?
- Recall@k asks whether the relevant chunks appear anywhere in the top k, which suits RAG when the LLM reads all k chunks and their order matters little.
- MRR averages one over the rank of the first relevant result, so it fits cases where only the top hit matters, such as a lookup that opens one document.
- NDCG rewards putting the most relevant results highest and supports graded relevance, but graded labels cost more to collect than binary ones.
Your synthetic eval set says retrieval is excellent, but users still complain about wrong answers; what is going on?
- Questions generated from a single chunk often reuse its wording, so they favour lexical and embedding matches and are easier than what real users type.
- Real users ask vague, multi-part or out-of-scope questions in their own vocabulary, and a set built chunk by chunk rarely covers those cases.
- Sample and label real queries from logs, cluster them by topic to find where retrieval underperforms, and add those cases to the eval set.
How would you test that your RAG system says it doesn’t know when the answer isn’t in the corpus?
- Add unanswerable questions to the eval set, both topics absent from the corpus and close look-alikes of covered ones, labelled with abstaining as the correct behaviour.
- Track the abstention rate on unanswerable questions alongside the false refusal rate on answerable ones, since a system that refuses everything aces the first metric.
- Read the failures to see whether wrong answers come from retrieval surfacing a near-miss chunk or from the generator ignoring its instruction to abstain.
How does HyDE work, and when could it make retrieval worse?
- HyDE asks the LLM to write a hypothetical answer and searches with that answer’s embedding, because an answer resembles the target passages more closely than a short question does.
- When the model invents wrong specifics, such as a made-up product name, retrieval is pulled towards plausible but wrong passages, and every query pays for an extra LLM call.
- Compare recall@k with and without it on your eval set, split by query type, since it may help vague questions while hurting ones with exact identifiers.
With a million-token context window, why not put every document in the prompt and skip retrieval?
- Cost and latency grow with input tokens on every request, so resending a large corpus is far slower and more expensive than retrieving a few chunks, even with prompt caching.
- Quality also drops as context grows, because models attend unevenly across long inputs and miss relevant facts buried among distractors, an effect often called context rot.
- Retrieval also lets you filter by user permissions and update documents cheaply; for a small, stable corpus stuffing can win, so test both on the same eval set.
When would you move from a single retrieval step to agentic RAG, and what does it cost you?
- Single-shot RAG searches once with the raw question, so it fails on multi-hop questions where the next search depends on what the first one found.
- An agent loop lets the model decide whether to search, rewrite the query, pick a source and check whether the results are enough before answering.
- Every extra step adds LLM calls, latency and less predictable behaviour, so cap iterations, log traces and compare task success against the simple pipeline before switching.
Rerankers and fine-tuning embedding models
Asked often
How would you build training data to fine-tune an embedding model for your domain?
- Collect query and relevant-passage pairs from search logs, clicks or support tickets, or generate synthetic questions from your own chunks with an LLM and filter out weak ones.
- Mine hard negatives by retrieving the top results for each query with BM25 or the current model and removing the known positive, which teaches finer distinctions than random negatives.
- Some mined negatives actually answer the query, so filter likely false negatives with a cross-encoder score threshold or by skipping the very top ranks before training.
Your fine-tuned retriever beat the base model on the eval set but does worse in production; what went wrong?
- The eval set may leak from training, for example synthetic questions drawn from the same chunks, so split by document rather than by pair to get an honest score.
- Training on a narrow set can erode the model’s general ability, so queries outside the training distribution get worse; check an out-of-domain set as well.
- Confirm the corpus was re-embedded with the new model and query prefixes match, then compare recall@k or NDCG against the old model on real logged queries.
What to study for Rerankers and fine-tuning embedding models
Vector database internals and index selection
Specialist topic or differentiator
After adding a metadata filter, vector search sometimes returns fewer than k results; why, and how would you fix it?
- With post-filtering, the ANN index returns its top candidates first and the filter then drops non-matching ones, so a selective filter can leave far fewer than k.
- Pre-filtering searches only matching vectors and keeps k results, but it either means exact search over the subset or walking a graph whose matching nodes may be poorly connected.
- Raise the candidate count or ef_search, use iterative index scans in pgvector, or partition by a common filter such as tenant, then measure recall at realistic filter selectivity.
Would you store embeddings in pgvector or a dedicated vector database, and how would you decide?
- pgvector keeps vectors beside your relational data, so filters, joins, transactions, backups and permissions all come from a database you already run.
- A dedicated vector database earns its extra system when the index outgrows one machine’s memory or you need sharding, heavy write throughput or built-in quantization.
- Estimate memory as vector count × dimensions × bytes per value plus index overhead, then load-test recall and p95 latency with your real filters before moving.
What to study for Vector database internals and index selection
Document parsing and ingestion for RAG: PDFs, tables, OCR
Specialist topic or differentiator
Retrieval finds the right PDF page, but answers about its tables are wrong; what do you check?
- Inspect the extracted text for that page first, since plain extraction often flattens a table into a stream of cells and loses which value sits under which header.
- Switch to a layout-aware parser that keeps table structure, and store each table as Markdown or HTML, optionally with a short summary embedded to help retrieval.
- Keep tables whole rather than splitting mid-row, repeat headers if a split is unavoidable, and measure the fix on a labelled set of table questions.
How would you choose between plain text extraction, layout-aware parsing and OCR for a mixed document set?
- Plain extraction reads the PDF’s embedded text layer quickly and cheaply, but it can scramble reading order in multi-column layouts and returns nothing for scanned pages.
- Layout-aware parsing detects headings, columns and tables at a higher cost, while OCR is required when pages are images, and it is slower and adds recognition errors.
- Route each document or page automatically, for example falling back to OCR when the text layer is empty or garbled, and spot-check output for each document type.
Your source documents change every day; how do you keep the index current without re-embedding everything?
- Store a content hash per document, and on each sync re-parse and re-embed only the documents whose hash changed, skipping everything else.
- Delete a changed or removed document’s old chunks by document ID before inserting new ones, otherwise stale chunks keep being retrieved and cited.
- Keep source, page, section and access labels on every chunk so permission changes propagate too, and monitor how far the index lags behind the sources.
Questions and answer points are written for SAADHO from each topic’s sources: question types interviewers ask, never leaked questions. How SAADHO is made