Skip to content

LLM-as-judge: building and validating judges

  • Evals
  • Asked in most interview loops
  • About 75 min to learn
  • 4 vetted resources
Build a plan that includes this

What this topic covers

An LLM judge is a prompt (and model) that grades another model's output so you can evaluate at scale without a human reading every trace. The hard part is not writing the prompt but proving the judge agrees with a trusted human: you collect expert pass/fail labels with written critiques, iterate the judge until it matches them, and keep re-checking because the criteria themselves shift once people look at real outputs. Interviews ask how you would build one for a given product, why binary verdicts beat 1-5 scales, how you measure judge-human agreement, and which biases to guard against (position, verbosity, self-preference, weak maths grading) and how to mitigate them.

Builds on

Leads to

You’re ready when you can

What interviewers check, from the resources below.

  • Explain why binary pass/fail with a written critique beats a 1-5 score
  • Describe how expert critiques become few-shot examples in the judge prompt
  • Explain how you'd measure judge-expert agreement and when to stop iterating
  • Name two common mistakes in judge prompts and how to fix them
  • Decide when a code assertion is better than an LLM judge
  • Compare direct scoring, pairwise comparison and reference-based judging
  • Name two known biases of LLM judges
  • Explain how you'd measure agreement between a judge and human labels
  • Define criteria drift and explain why it breaks “write the rubric first, then grade”
  • Describe how human grades are used to pick the evaluators that agree best

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Using LLM-as-a-Judge For Evaluation: A Complete Guide (opens in a new tab)

    hamel.dev · Hamel Husain · 35 min · Free · checked 21 Oct 2026

    Why
    “How would you evaluate this feature?” usually ends at an LLM judge, and interviewers want a process for trusting it, not just a grading prompt.
    What to read
    Read “Step 3: Direct The Domain Expert to Make Pass/Fail Judgments with Critiques” through “Step 5: Build Your LLM as A Judge, Iteratively” (including “Mistakes I've noticed in LLM judge prompts”), then “Step 7: Create More Specialized LLM Judges (if needed)” and “Recap of Critique Shadowing”.
    What to skip
    “Step 2: Create a Dataset” and “Step 6: Perform Error Analysis” (covered in evals-error-analysis), the Andrew Ng video, and “Resources”; skim the “FAQ” only for the judge-model and validation questions.

  2. Free alternative

    Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge) (opens in a new tab)

    eugeneyan.com · Eugene Yan · 30 min · Free · checked 20 Oct 2026

    Why
    When you propose an LLM judge, the follow-up is “how do you know the judge is right?” This answers it.
    What to read
    Read “Key considerations”, “Techniques for prompting” and “Aligning LLM-evaluators to our criteria”.
    What to skip
    Finetuning evaluator models, and the long reference list (it's a 49-minute read in full).

  3. Go deeper

    Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences (opens in a new tab)

    arXiv · Shreya Shankar, J.D. Zamfirescu-Pereira, Björn Hartmann, Aditya G. Parameswaran, Ian Arawjo · 40 min · Free · checked 21 Oct 2026

    Why
    The research behind “validate your judge against human grades”, and the source of criteria drift, a term interviewers increasingly expect you to know.
    What to read
    Read “1 Introduction”, “3.1 EvalGen Workflow”, “7.3 Alignment is Iterative, Criteria- and Implementation-specific” (especially “7.3.1 Criteria drift”) and “8.1 Implications of Criteria Drift for LLM Evaluation Assistants”.
    What to skip
    “4.2 Selecting Assertions & Eliciting Grades”, “5 Algorithm Evaluation”, the rest of the user-study findings and the appendices unless you're building eval tooling.

  4. Go deeper

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (opens in a new tab)

    arXiv (NeurIPS 2023 Datasets and Benchmarks) · Lianmin Zheng et al. · 40 min · Free · checked 21 Oct 2026

    Why
    The paper that named the judge biases (position, verbosity, self-enhancement) interviewers ask you to list and mitigate.
    What to read
    Read “3 LLM as a Judge” (“3.1 Types of LLM-as-a-Judge” through “3.4 Addressing limitations”) and “4.2 High agreement between GPT-4 and humans”.
    What to skip
    “2 MT-Bench and Chatbot Arena” beyond the overview, “5 Human Preference Benchmark and Standardized Benchmark”, and the appendices except a glance at “Appendix A: Prompt templates”.

Companies that test Evals most

Put LLM-as-judge in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan