# RAG interview questions

> 25 RAG interview questions with what a strong answer covers.

Canonical: https://saadho.arushk.tech/questions/rag

What interviews ask about retrieval-augmented generation: chunking and ingestion, embeddings and vector search, hybrid search and reranking, and how you measure whether retrieval and answers are any good. 25 questions across 7 topics.

## Retrieval for RAG: chunking, hybrid search, reranking

[What to study for this topic](https://saadho.arushk.tech/topics/rag-retrieval)

**Retrieval keeps missing answers that are clearly in the documents; what would you try first?**

- Check chunking: chunks cut mid-thought or stripped of their document title lose context, so use structure-aware splits, overlap or prepended context.
- Add BM25 beside embeddings, since keywords catch exact names and codes while embeddings catch paraphrases, and fuse the lists with reciprocal rank fusion.
- Measure every change with recall@k on a labelled or synthetic question set, so you know retrieval improved rather than just changed.

**Where does a reranker sit in a RAG pipeline, and what does it cost you?**

- First-stage retrieval cheaply pulls a wide candidate set with BM25 and vector search, favouring recall over precision.
- A cross-encoder reranker reads the query and each candidate together, scoring relevance more accurately than comparing separate embeddings.
- It adds latency and cost per candidate, so you cap the candidate count and pass only the top few chunks to the LLM.

**How would you choose a chunk size, and what goes wrong at each extreme?**

- Small chunks give focused embeddings and little noise, but they can split an answer across chunks or lose context such as which product a sentence is about.
- Large chunks keep context but mix several topics into one embedding, which weakens its match to a specific query and spends more of the prompt on irrelevant text.
- Sweep a few sizes and overlaps on your eval set, measuring recall@k and end-to-end answer quality, since the best size depends on your documents and queries.

**How does reciprocal rank fusion merge BM25 and vector results, and why not just add the scores?**

- RRF gives each document the sum over result lists of 1/(k + rank), with k commonly 60, so documents ranked well by both retrievers rise to the top.
- It uses ranks only, so you never have to put unbounded, query-dependent BM25 scores and cosine similarities onto a common scale before combining them.
- The cost is discarding score magnitudes, so a tuned, normalised weighted fusion can sometimes beat it; compare both on recall@k or NDCG before choosing.

**Users ask about the current version of a policy, but retrieval returns chunks from every old version too; how would you fix it?**

- Embeddings can’t tell versions apart, because old and new chunks say almost the same thing, so similarity alone has no reason to prefer the current one.
- Attach version, effective date and source as metadata at ingestion, then filter or boost on them, extracting the filter from the question with rules or an LLM.
- Label a set of version- and date-specific questions and measure how often the top-k chunks come from the correct version, before and after the change.

## Embeddings and vector search

[What to study for this topic](https://saadho.arushk.tech/topics/embeddings-vector-search)

**When would you use cosine similarity versus dot product for vector search?**

- Cosine compares direction only and ignores vector length; dot product also rewards magnitude, which some models are trained to use.
- On L2-normalised vectors cosine and dot product give identical rankings, so you can normalise once and use the cheaper dot product.
- Use the metric the embedding model was trained with; mismatching it quietly hurts retrieval quality without raising any error.

**How does HNSW find nearest neighbours fast, and which knobs trade recall for latency?**

- HNSW builds a layered graph: sparse upper layers for long hops, dense lower layers for fine search, walked greedily from the top.
- M sets links per node, trading memory and build time for recall; ef_construction sets how carefully the graph is built.
- ef_search sets the candidate list size at query time: higher gives better recall but slower queries, tuned against exact search results.

**What is the difference between symmetric and asymmetric semantic search, and why does it change your model choice?**

- Symmetric search compares texts of similar form, like finding duplicate questions; asymmetric search matches a short query to a longer passage that answers it.
- Models trained for asymmetric retrieval learn to place a question near its answer even though the two look different, while a symmetric model may surface similar questions instead.
- Many asymmetric models expect different prefixes or instructions for queries and documents, and leaving them out quietly lowers recall, so follow the model card and check on your eval set.

**When is exact nearest-neighbour search good enough, and what do you give up by moving to an approximate index?**

- Flat search compares the query with every vector, so it always returns the true neighbours, but its cost grows linearly with corpus size and dimension.
- Approximate indexes such as HNSW or IVF visit only a fraction of the vectors, trading a little recall and extra memory or build time for much lower latency.
- Run sample queries through both, measure the approximate index’s recall@k against the exact results, and pick the cheapest settings that meet your recall and latency targets.

**You want to switch to a newer embedding model; what does the migration involve?**

- Vectors from different models live in unrelated spaces, so the whole corpus must be re-embedded; mixing old and new vectors in one index makes similarities meaningless.
- Dimension, similarity metric, maximum input length and query prefixes may all change, which affects index configuration, chunk size and storage cost.
- Compare both models on your own labelled retrieval set with recall@k or NDCG, since leaderboard rankings may not hold on your domain, then build a new index and switch over.

## Evaluating RAG: retrieval metrics and faithfulness

[What to study for this topic](https://saadho.arushk.tech/topics/rag-evaluation)

**How would you evaluate retrieval and generation separately in a RAG system?**

- Score retrieval with recall@k, MRR or NDCG against questions labelled with the chunks that answer them; context precision also rewards good rank order.
- Score generation for faithfulness, supported claims divided by total claims, and answer relevance, usually with an LLM judge or NLI model.
- With no labels, generate synthetic questions from your own chunks, then hand-check a sample so the set resembles real user questions.

**Faithfulness went up after your change; does that mean the product got better?**

- Not necessarily: faithfulness only says claims are backed by the context, so shorter or more hedged answers can score higher.
- Check context recall, which needs a reference answer, and answer relevance too, since a faithful answer built on the wrong chunks still fails users.
- The LLM judge inside these metrics can be wrong, so spot-check its claim splits and verdicts against human labels before trusting the trend.

**When would you report recall@k, MRR or NDCG for a RAG retriever?**

- Recall@k asks whether the relevant chunks appear anywhere in the top k, which suits RAG when the LLM reads all k chunks and their order matters little.
- MRR averages one over the rank of the first relevant result, so it fits cases where only the top hit matters, such as a lookup that opens one document.
- NDCG rewards putting the most relevant results highest and supports graded relevance, but graded labels cost more to collect than binary ones.

**Your synthetic eval set says retrieval is excellent, but users still complain about wrong answers; what is going on?**

- Questions generated from a single chunk often reuse its wording, so they favour lexical and embedding matches and are easier than what real users type.
- Real users ask vague, multi-part or out-of-scope questions in their own vocabulary, and a set built chunk by chunk rarely covers those cases.
- Sample and label real queries from logs, cluster them by topic to find where retrieval underperforms, and add those cases to the eval set.

**How would you test that your RAG system says it doesn’t know when the answer isn’t in the corpus?**

- Add unanswerable questions to the eval set, both topics absent from the corpus and close look-alikes of covered ones, labelled with abstaining as the correct behaviour.
- Track the abstention rate on unanswerable questions alongside the false refusal rate on answerable ones, since a system that refuses everything aces the first metric.
- Read the failures to see whether wrong answers come from retrieval surfacing a near-miss chunk or from the generator ignoring its instruction to abstain.

## Advanced RAG: query rewriting, reranking, agentic RAG, long context vs RAG

[What to study for this topic](https://saadho.arushk.tech/topics/advanced-rag)

**How does HyDE work, and when could it make retrieval worse?**

- HyDE asks the LLM to write a hypothetical answer and searches with that answer’s embedding, because an answer resembles the target passages more closely than a short question does.
- When the model invents wrong specifics, such as a made-up product name, retrieval is pulled towards plausible but wrong passages, and every query pays for an extra LLM call.
- Compare recall@k with and without it on your eval set, split by query type, since it may help vague questions while hurting ones with exact identifiers.

**With a million-token context window, why not put every document in the prompt and skip retrieval?**

- Cost and latency grow with input tokens on every request, so resending a large corpus is far slower and more expensive than retrieving a few chunks, even with prompt caching.
- Quality also drops as context grows, because models attend unevenly across long inputs and miss relevant facts buried among distractors, an effect often called context rot.
- Retrieval also lets you filter by user permissions and update documents cheaply; for a small, stable corpus stuffing can win, so test both on the same eval set.

**When would you move from a single retrieval step to agentic RAG, and what does it cost you?**

- Single-shot RAG searches once with the raw question, so it fails on multi-hop questions where the next search depends on what the first one found.
- An agent loop lets the model decide whether to search, rewrite the query, pick a source and check whether the results are enough before answering.
- Every extra step adds LLM calls, latency and less predictable behaviour, so cap iterations, log traces and compare task success against the simple pipeline before switching.

## Rerankers and fine-tuning embedding models

[What to study for this topic](https://saadho.arushk.tech/topics/rerankers-embedding-finetuning)

**How would you build training data to fine-tune an embedding model for your domain?**

- Collect query and relevant-passage pairs from search logs, clicks or support tickets, or generate synthetic questions from your own chunks with an LLM and filter out weak ones.
- Mine hard negatives by retrieving the top results for each query with BM25 or the current model and removing the known positive, which teaches finer distinctions than random negatives.
- Some mined negatives actually answer the query, so filter likely false negatives with a cross-encoder score threshold or by skipping the very top ranks before training.

**Your fine-tuned retriever beat the base model on the eval set but does worse in production; what went wrong?**

- The eval set may leak from training, for example synthetic questions drawn from the same chunks, so split by document rather than by pair to get an honest score.
- Training on a narrow set can erode the model’s general ability, so queries outside the training distribution get worse; check an out-of-domain set as well.
- Confirm the corpus was re-embedded with the new model and query prefixes match, then compare recall@k or NDCG against the old model on real logged queries.

## Vector database internals and index selection

[What to study for this topic](https://saadho.arushk.tech/topics/vector-db-internals)

**After adding a metadata filter, vector search sometimes returns fewer than k results; why, and how would you fix it?**

- With post-filtering, the ANN index returns its top candidates first and the filter then drops non-matching ones, so a selective filter can leave far fewer than k.
- Pre-filtering searches only matching vectors and keeps k results, but it either means exact search over the subset or walking a graph whose matching nodes may be poorly connected.
- Raise the candidate count or ef_search, use iterative index scans in pgvector, or partition by a common filter such as tenant, then measure recall at realistic filter selectivity.

**Would you store embeddings in pgvector or a dedicated vector database, and how would you decide?**

- pgvector keeps vectors beside your relational data, so filters, joins, transactions, backups and permissions all come from a database you already run.
- A dedicated vector database earns its extra system when the index outgrows one machine’s memory or you need sharding, heavy write throughput or built-in quantization.
- Estimate memory as vector count × dimensions × bytes per value plus index overhead, then load-test recall and p95 latency with your real filters before moving.

## Document parsing and ingestion for RAG: PDFs, tables, OCR

[What to study for this topic](https://saadho.arushk.tech/topics/document-ingestion)

**Retrieval finds the right PDF page, but answers about its tables are wrong; what do you check?**

- Inspect the extracted text for that page first, since plain extraction often flattens a table into a stream of cells and loses which value sits under which header.
- Switch to a layout-aware parser that keeps table structure, and store each table as Markdown or HTML, optionally with a short summary embedded to help retrieval.
- Keep tables whole rather than splitting mid-row, repeat headers if a split is unavoidable, and measure the fix on a labelled set of table questions.

**How would you choose between plain text extraction, layout-aware parsing and OCR for a mixed document set?**

- Plain extraction reads the PDF’s embedded text layer quickly and cheaply, but it can scramble reading order in multi-column layouts and returns nothing for scanned pages.
- Layout-aware parsing detects headings, columns and tables at a higher cost, while OCR is required when pages are images, and it is slower and adds recognition errors.
- Route each document or page automatically, for example falling back to OCR when the text layer is empty or garbled, and spot-check output for each document type.

**Your source documents change every day; how do you keep the index current without re-embedding everything?**

- Store a content hash per document, and on each sync re-parse and re-embed only the documents whose hash changed, skipping everything else.
- Delete a changed or removed document’s old chunks by document ID before inserting new ones, otherwise stale chunks keep being retrieved and cited.
- Keep source, page, section and access labels on every chunk so permission changes propagate too, and monitor how far the index lags behind the sources.

## Related

- [All topics](https://saadho.arushk.tech/library)
- [Build a plan](https://saadho.arushk.tech/start)