Skip to content

Statistics for LLM evals

  • Evals
  • Specialist topic or differentiator
  • About 1 h 30 min to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

Eval scores are estimates from a sample of questions, so every score needs an error bar before you act on it. Interviewers ask “is this +2 points real?” to see whether you can reason about noise rather than just read a leaderboard. A strong answer computes a standard error and confidence interval, uses a paired test when two models answer the same items, clusters errors when questions share a source, and runs a power calculation to size the eval set. It also explains why normal-approximation intervals are too narrow on small or near-perfect eval sets and what to use instead.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Can compute a standard error and 95% interval for a pass rate
  • Uses a paired difference when two models answer the same items, and says why it’s tighter
  • Knows when to cluster standard errors and which way the naive error is wrong
  • Can estimate how many items are needed to detect a given difference
  • Explains why normal-approximation intervals mislead on small or near-perfect sets
  • Can state the five recommendations from memory
  • Explains treating eval questions as a sample from a larger question pool
  • Knows two ways to reduce within-question noise: resampling and token probabilities
  • Can explain why CLT intervals come out too narrow on small eval sets
  • Knows what goes wrong when scores are all pass or all fail

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Applying Statistics to LLM Evaluations (opens in a new tab)

    Deep (Learning) Focus · Cameron R. Wolfe · 60 min · Free · checked 21 Oct 2026

    Why
    “Is this +2 points real?” comes up whenever you report an eval result. This post builds the statistics from scratch and then applies them to the exact eval cases you’ll be asked about.
    What to read
    Read “Basic Statistics for LLM Evaluations” if you’re rusty, then all of “A Statistical Approach to LLM Evaluations” (standard errors, clustered errors, reducing variance, model comparisons). Finish with “Power Analysis for LLM Evals” and the small-sample section under “More Topics to Explore in Statistics”.
    What to skip
    Skim “Quantifying Variance in Evaluation Benchmarks” and “Signal and Noise”; they’re about benchmark design, not your own evals.

  2. Free alternative

    A statistical approach to model evaluations (opens in a new tab)

    Anthropic · 20 min · Free · checked 21 Oct 2026

    Why
    A short, citable summary of five practices for putting error bars on evals. Good to quote when an interviewer asks how a lab reports eval results.
    What to read
    Read all five recommendations: use the Central Limit Theorem, cluster standard errors, reduce variance within questions, analyse paired differences, and use power analysis.
    What to skip
    Nothing to skip; the linked paper (arXiv 2411.00640) has the formulas if you want them.

  3. Go deeper

    Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints (opens in a new tab)

    arXiv · Sam Bowyer, Laurence Aitchison, Desi R. Ivanova · 45 min · Free · checked 21 Oct 2026

    Why
    Most product eval sets are small, which is exactly where the usual error bars break. This ICML 2025 position paper shows how badly and what to use instead.
    What to read
    Read section 1 “Introduction”, then section 3 “Failures of the CLT for LLM Evals”, at least 3.1 (independent questions) and 3.4 (paired model comparison). Note the recommended replacements: the Wilson score interval or a Bayesian Beta posterior, and the bayes_evals Python library.
    What to skip
    Skip section 4 “Alternative Views” on a first pass, and the appendices except A if you want the interval formulas.

Companies that test Evals most

Put Statistics for LLM evals in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan