Evals and error analysis
- Evals
- Asked in most interview loops
- About 2 h 30 min to learn
- 4 vetted resources
What this topic covers
Evals are how you know an LLM product got better rather than just different. The core loop is: log real traces, read them, group the failures into categories, then write targeted checks (code assertions first, LLM-as-judge only where needed) and keep the judge aligned with human labels. Many AI engineering loops now include an evals discussion or take-home.
Leads to
- LLM-as-judge: building and validating judges
- GenAI system design
- Evaluating RAG: retrieval metrics and faithfulness
- Eval datasets, benchmarks and evals in CI
- Agent reliability: evals, failure modes and observability
- Take-home projects and eval take-homes
- LLM observability and feedback loops
- Design an eval: the live exercise
- AI product sense and scoping
You’re ready when you can
What interviewers check, from the resources below.
- Describe the three levels of evaluation and when each runs
- Explain why you read traces before writing any LLM judge
- Give one example of a scoped, assertion-style test for an LLM feature
- Walk through error analysis: sample traces → open coding → failure categories
- Explain bottom-up vs top-down analysis
- Explain criteria drift and how you'd keep a judge trustworthy
- Name two retrieval failure modes and a mitigation for each
- Explain why RAG is preferred over fine-tuning for adding knowledge
- Sketch a RAG system with its offline (indexing) and online (query) halves
- Compare direct scoring, pairwise comparison and reference-based judging
What to study, in order
Each link opened and checked by hand; the date says when.
Start here
Your AI Product Needs Evals (opens in a new tab)
hamel.dev · Hamel Husain · 25 min · Free · checked 20 Oct 2026
- Why
- The reference text for how practitioners think about evals; interviewers often borrow its vocabulary.
- What to read
- Read “The Types Of Evaluation” (unit tests, human & model eval, A/B testing) using the Lucy case study.
- What to skip
- “Eval Systems Unlock Superpowers For Free” on a first pass.
Free alternative
A Field Guide to Rapidly Improving AI Products (opens in a new tab)
hamel.dev · Hamel Husain · 35 min · Free · checked 20 Oct 2026
- Why
- Error analysis is the step most candidates skip, and the one evals interviews probe hardest.
- What to read
- Read sections 1 (error analysis), 4 (synthetic data with zero users) and 5 (trust in evals).
- What to skip
- Section 6 on roadmaps: useful at work, rarely asked.
Go deeper
Patterns for Building LLM-based Systems & Products (opens in a new tab)
eugeneyan.com · Eugene Yan · 25 min · Free · checked 20 Oct 2026
- Why
- A widely cited map of LLM system patterns; a good frame for a system design answer.
- What to read
- Read only “Retrieval-Augmented Generation: To add knowledge” (and “Evals” if you're on the evals topic).
- What to skip
- Fine-tuning, caching and defensive UX for now. The full piece is a 66-minute read.
Go deeper
Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge) (opens in a new tab)
eugeneyan.com · Eugene Yan · 30 min · Free · checked 20 Oct 2026
- Why
- When you propose an LLM judge, the follow-up is “how do you know the judge is right?” This answers it.
- What to read
- Read “Key considerations”, “Techniques for prompting” and “Aligning LLM-evaluators to our criteria”.
- What to skip
- Finetuning evaluator models, and the long reference list (it's a 49-minute read in full).
Companies that test Evals most
Put Evals and error analysis in your plan
Tell us your interview date and hours a day; it lands on the right day, in the right order.