Skip to content

Eval datasets, benchmarks and evals in CI

  • Evals
  • Asked in most interview loops
  • About 2 h to learn
  • 5 vetted resources
Build a plan that includes this

What this topic covers

Once error analysis has told you what breaks, you need a durable test set and a place to run it. This topic covers how an eval dataset is built (inputs, expected outputs or pass/fail criteria, edge cases drawn from real failures), how big it needs to be, how it goes stale and gets refreshed from production traces, and how to grade it cheaply enough to run on every pull request. It also covers the split between public benchmarks (useful for picking a model, useless for your product) and domain-specific evals for RAG, agents and multi-turn chat. Interviewers ask how you would stop a prompt or model change from regressing, what your CI eval looks like, what it costs to run, and how you decide a case belongs in the gold set.

Builds on

Leads to

You’re ready when you can

What interviewers check, from the resources below.

  • Contrast a CI eval set (small, curated, cheap deterministic checks) with production monitoring (sampled traces, async LLM judges) and say why they differ
  • Describe how a failure found in production becomes a new CI test case
  • Explain why you evaluate retrieval with Recall@k / MRR before you evaluate generation in a RAG system
  • Outline how to test an agent: end-to-end task success first, then step-level checks on tool choice and parameters
  • Say what you do when every example in the gold set passes
  • Turn a vague goal (“classify sentiment well”) into a measurable target with a number and a dataset
  • Explain why an eval set should mirror the real input distribution and include deliberate edge cases
  • Choose between exact match, embedding similarity, ROUGE and an LLM grader for a given task and justify it
  • State why the grading model should differ from the model being graded
  • Write a JSONL eval file with an input and an ideal answer per line

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    AI Evals: Everything You Need to Know (opens in a new tab)

    Hamel's Blog · Hamel Husain, Shreya Shankar · 45 min · Free · checked 21 Oct 2026

    Why
    This FAQ answers the exact follow-ups interviewers use after “how do you evaluate it?”: what runs in CI, what runs in production, and how the two feed each other.
    What to read
    Read “How are evaluations used differently in CI/CD vs. monitoring production?”, then the “Production & Deployment” and “Domain-Specific Applications” sections, especially “How should I approach evaluating my RAG system?”, “How do I debug multi-turn conversation traces?” and “How do I evaluate agentic workflows?”.
    What to skip
    The opening sections on error analysis and annotation if you already did the evals-error-analysis topic; the “Tools & Infrastructure” section can be skimmed.

  2. Free alternative

    Define success criteria and build evaluations (opens in a new tab)

    Claude Developer Platform docs · Anthropic · 30 min · Free · checked 21 Oct 2026

    Why
    Vendor docs that lay out the three grading styles (code, model, human) with concrete example evals you can quote in an interview.
    What to read
    Read “Define your success criteria”, “Build evaluations”, “Eval design principles” and “Example evals”.
    What to skip
    The “Common success criteria” table once you have the idea; the later prompt-engineering pages linked from the bottom.

  3. Go deeper

    Building an eval (opens in a new tab)

    openai/evals (GitHub) · OpenAI · 40 min · Free · checked 21 Oct 2026

    Why
    Shows the mechanics of a real eval harness: a JSONL dataset, a template that grades it, a registry entry, a versioned run. Having done this once makes “what does your CI eval look like” easy to answer.
    What to read
    Read “Formatting your data”, “Registering the eval” and “Running the eval”, then write a 20-line JSONL eval of your own and run it with the Match template.
    What to skip
    “Criteria for contributing an eval”; the model-graded workflow unless you have time, since llm-as-judge is its own topic.

  4. Go deeper

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (opens in a new tab)

    arXiv · Carlos E. Jimenez et al. · 35 min · Free · checked 22 Oct 2026

    Why
    SWE-bench is the benchmark coding agents are judged by, and its design, real issues checked by fail-to-pass tests, is a model answer for “how would you build an execution-based eval?”
    What to read
    Read §2.1 with Figure 2 (how issues and tests were mined and filtered), §2.2 on the task format and §2.3 on what makes it hard. Then read §2.4 on SWE-bench Lite, §4.1 (BM25 versus oracle retrieval) and Appendix A.3–A.4 on execution-based validation.
    What to skip
    Skip §3 on SWE-Llama, §5.1 qualitative analysis, §6 related work and Appendices B–D.

  5. Go deeper

    Language Model Evaluation Harness (opens in a new tab)

    GitHub · EleutherAI · 45 min · Free · checked 22 Oct 2026

    Why
    Most published benchmark numbers for open models come from lm-evaluation-harness, so knowing how a task is defined (prompt template, few-shot count, answer extraction) explains why the same model gets different scores in different papers.
    What to read
    Read lm_eval/tasks/gsm8k/gsm8k.yaml (doc_to_text, num_fewshot, generation_kwargs, the strict-match and flexible-extract filters) next to lm_eval/tasks/hellaswag/hellaswag.yaml for a multiple-choice task. Then read docs/new_task_guide.md. Run a small model, for example lm_eval --model hf --model_args pretrained=EleutherAI/pythia-160m --tasks hellaswag --limit 100, and look at the reported stderr.
    What to skip
    Skip the backend sections (NeMo, Megatron, OpenVINO, ONNX, Windows ML), multi-GPU setup and steered models.

Companies that test Evals most

Put Eval datasets, benchmarks and evals in CI in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan