A/B testing and product sense for AI features
- Evals
- Asked often
- About 1 h 30 min to learn
- 3 vetted resources
What this topic covers
Offline evals tell you whether a change is plausibly better. An online A/B test tells you whether users are actually better off. Interviewers use this topic to test product sense: can you pick one success metric plus guardrails (acceptance or edit rate, regenerations, latency, cost per request, refusals), size the experiment, and read the result honestly? A strong answer covers dark launches and shadow tests before exposing users, how non-deterministic outputs add variance and so raise the sample size you need, and the peeking problem. It also knows when a simpler rule or classifier would beat an LLM.
You’re ready when you can
What interviewers check, from the resources below.
- Pick one success metric and three guardrail metrics for a given AI feature
- Explain dark-mode, shadow and 0-1 experiments and when to use each
- Say why shadow experiments reduce variance, and which metrics they can’t measure
- Name the telemetry needed to separate user edits from model edits
- Write a one-sentence hypothesis with a primary metric for an LLM change
- List the logging you need before launch (variant, latency, tokens, user feedback)
- Explain how you would weigh a quality win against a cost or latency loss
- Explain baseline rate, minimum detectable effect, power and significance in one line each
- Show roughly how halving the detectable effect changes the sample size (about four times)
- Explain why stopping a test early on significance inflates false positives
What to study, in order
Each link opened and checked by hand; the date says when.
Start here
How to Evaluate LLMs: A Complete Metric Framework (opens in a new tab)
Microsoft Research (Experimentation Platform) · 40 min · Free · checked 21 Oct 2026
- Why
- “How would you A/B test this AI feature?” needs a metric framework and a rollout sequence. This is both, from the team that runs experiments for Microsoft products.
- What to read
- Read the four metric families (“GPU Utilization”, “Responsible AI”, “Performance Metrics”, “Utility Metrics”), then “Data Requirements” and “Running A/B tests” (dark mode, 0-1, shadow and 1-N experiments).
- What to skip
- Skip the appendix metric lists and the collaboration-productivity detail unless your target role is in productivity software.
Free alternative
Beyond prompts: A data-driven approach to LLM optimization (opens in a new tab)
Statsig · Anna Yoon · 25 min · Free · checked 21 Oct 2026
- Why
- It walks the full experiment lifecycle for an LLM change in plain terms. That is a good scaffold for a product-sense answer.
- What to read
- Read “Mechanics of A/B testing for LLMs”, “Key metrics to track” and all of “Experiment design and statistical rigor”, from hypothesis through rolling out or iterating.
- What to skip
- Skim the case studies; they report results without the statistics. Skip the demo request and the related links.
Go deeper
Sample Size Calculator (Evan’s Awesome A/B Tools) (opens in a new tab)
evanmiller.org · Evan Miller · 30 min · Free · checked 21 Oct 2026
- Why
- You may be asked to size a test on the spot. Playing with the inputs builds intuition for how a small effect or a noisy metric blows up the sample you need.
- What to read
- Enter a baseline acceptance rate (say 30%) and try relative minimum detectable effects of 10%, 5% and 2% at 80% power. Then follow the “See also” link to “How Not To Run an A/B Test” and read it.
- What to skip
- Skip the other calculators and the app promotion.
Companies that test Evals most
Put A/B testing and product sense for AI features in your plan
Tell us your interview date and hours a day; it lands on the right day, in the right order.