Evaluating RAG: retrieval metrics and faithfulness
- RAG
- Asked in most interview loops
- About 80 min to learn
- 3 vetted resources
What this topic covers
A RAG system can fail in two separate places: the retriever can miss or mis-rank the right chunks, and the generator can say things the chunks don't support. Evaluating RAG means scoring those two stages separately. Retrieval is scored with classic IR metrics (recall@k, MRR, MAP, NDCG) against a labelled or synthetic question set; generation is scored for faithfulness (is every claim backed by the context?) and answer relevance, usually with an LLM judge or a small NLI classifier. Interviewers ask you to define these metrics, explain which one to pick for a given product, and describe how you'd build the eval set in the first place when no labels exist. Expect the follow-up: 'the metric went up, is the product better?'
You’re ready when you can
What interviewers check, from the resources below.
- Define faithfulness as supported claims over total claims and walk through a 0.5 example
- Explain why context recall needs a reference while faithfulness does not
- State what context precision rewards that plain recall@k ignores (rank order)
- Name one failure mode of using an LLM as the judge inside these metrics
- Compute recall@k, MRR and NDCG@k by hand for a ranked list of 5 results
- Explain which metric to pick when only the top result matters vs. when the whole page matters
- State why NDCG needs graded relevance and what that costs in labelling
- Describe how to get the relevance labels for a RAG corpus when none exist
- Describe how to bootstrap a retrieval eval set from chunks with no user data
- Explain why the author replaced thumbs up/down with a yes/no 'was this answered' question
What to study, in order
Each link opened and checked by hand; the date says when.
Start here
List of available metrics (opens in a new tab)
Ragas docs · Ragas (Exploding Gradients) · 25 min · Free · checked 21 Oct 2026
- Why
- Ragas's metric names (faithfulness, context precision, context recall) are the shared vocabulary interviewers use when they ask how you'd evaluate a RAG system.
- What to read
- Start at the “Retrieval Augmented Generation” list, then open the three linked pages Faithfulness (read “How It's Calculated” and the Einstein example), Context Precision (the intro formula and the “Context Precision” / “Context Utilization” examples) and Context Recall (the intro and first “Example”). Note which ones need a reference answer.
- What to skip
- The “Nvidia Metrics”, “SQL”, “Agents or Tool use cases” sections, the Legacy Metrics API blocks, and the ID-based / non-LLM variants.
Free alternative
Evaluation Measures in Information Retrieval (opens in a new tab)
Pinecone Learning Center · Laura Carnevali · 35 min · Free · checked 21 Oct 2026
- Why
- Before anyone asks about LLM-judged metrics they ask you to define recall@k, MRR and NDCG on a whiteboard; this is the cleanest walkthrough with a toy dataset and code.
- What to read
- Read “Metrics in Information Retrieval” through “Normalized Discounted Cumulative Gain (NDCG@K)”, including each metric's “Pros and Cons”. Work the eight-image “Cats in Boxes” example by hand before reading the Python.
- What to skip
- “References” and “Nomenclature”.
Go deeper
Systematically Improving Your RAG (opens in a new tab)
jxnl.co · Jason Liu · 25 min · Free · checked 21 Oct 2026
- Why
- Turns metrics into a process: synthetic question sets, measuring retrieval precision/recall per query cluster, and feedback wording, which is what senior-level 'how would you improve this RAG system' questions are really probing.
- What to read
- Read “Start with Synthetic Data”, “Utilize Metadata”, “Use Both Full-Text Search and Vector Search”, “Implement Clear User Feedback Mechanisms”, “Cluster and Model Topics” and “Balance Latency and Performance”. Pay attention to the recall numbers he reports for full-text vs. embeddings on different corpora.
- What to skip
- “Wrapping Up” and “Want to learn more?” (course promo).
Companies that test RAG most
Put Evaluating RAG in your plan
Tell us your interview date and hours a day; it lands on the right day, in the right order.