Skip to content

Design an eval: the live exercise

  • Evals
  • Asked in most interview loops
  • About 2 h to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

Many AI engineering loops now include a round where you’re handed a vague product goal, such as “a support assistant that helps customers”, and asked how you’d know it’s helping and not hurting. Interviewers use it to see whether you can turn a fuzzy quality bar into concrete test cases before touching a prompt. A strong answer starts from the user’s job and the costly failures, sketches a small labelled set of 20 or so cases that covers normal, edge and adversarial inputs, and picks a grader you’ve checked against human labels. It then ties offline scores to production signals like escalations, resolution and complaints, and states a clear rule for when to ship, hold or roll back. When offline and online disagree, you explain why the production signal usually wins and how you’d find the gap in your eval set.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Can write a small labelled set that deliberately includes real failure cases
  • Uses binary pass/fail labels and one grader per criterion, not one catch-all judge
  • Checks a judge against human labels with precision, recall on fails and agreement
  • Can explain how sample size limits what an eval result can prove
  • Names failure modes the spec didn’t mention before writing any cases
  • Writes rubric criteria precise enough that two graders would agree
  • Says how the eval set will track live traffic rather than a stale curated set
  • Can explain why a passing offline score can coexist with worse production outcomes
  • Can describe eval-driven development: success criteria written before the feature
  • Explains why an automated judge needs ongoing human sampling after launch

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Product Evals in Three Simple Steps (opens in a new tab)

    eugeneyan.com · Eugene Yan · 30 min · Free · checked 21 Oct 2026

    Why
    The eval-design round asks you to build a working eval from nothing, out loud. This is a compact, practical recipe for exactly that: labels first, then a grader you’ve checked, then a harness that runs on every change.
    What to read
    Read all three steps: “First, label some data”, “Then, align our LLM-evaluator” and “Finally, run our eval harness with each change”. Afterwards, pick a support assistant and write 20 pass/fail cases on paper before any prompt, then say which grader you’d trust for each.
    What to skip
    Skim “Additional reading” and the course promotion; the three steps are the whole lesson.

  2. Free alternative

    The AI Engineer Interview Is Broken: Stop Testing Implementation, Start Probing Eval-Design (opens in a new tab)

    TianPan.co · Tian Pan · 20 min · Free · checked 21 Oct 2026

    Why
    This is the interviewer’s side of the round: what they hand you, and what separates a strong eval from a weak one. Knowing the scoring lets you aim your answer.
    What to read
    Read “What an eval-design round actually looks like” closely and note its four scoring checks. Then read “The debugging-replay round catches the taste signal” and the anecdote about a high CI score while the support queue grew.
    What to skip
    Skip “What to drop, and why dropping it is harder than adding rounds”; it’s advice for hiring managers, not candidates.

  3. Go deeper

    An LLM-as-Judge Won't Save The Product—Fixing Your Process Will (opens in a new tab)

    eugeneyan.com · Eugene Yan · 15 min · Free · checked 21 Oct 2026

    Why
    Interviewers often push past the test set and ask how the eval stays honest after launch. This short piece gives you the loop to describe: look at data, form a hypothesis, measure, and keep humans in it.
    What to read
    Read the parts on evals as the scientific method, eval-driven development, and why human oversight is still needed even with an automated judge. Use it to frame your ship and rollback rule as one turn of that loop.
    What to skip
    Skip the closing course promotion.

Companies that test Evals most

Put Design an eval in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan