Skip to content

Classification metrics

  • ML fundamentals
  • Asked often
  • About 60 min to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

How to judge a classifier beyond accuracy: precision, recall and F1 from a confusion matrix, how the decision threshold trades them off, what a ROC curve and AUC actually measure, and whether predicted probabilities can be trusted (calibration). In Applied LLM interviews this shows up when you evaluate guardrails, retrieval filters, LLM-judge agreement or any binary decision a model makes. Expect to compute precision and recall from a small table, pick the right metric for an imbalanced or cost-asymmetric case, explain AUC in one sentence, and say when a high AUC still hides a badly calibrated model.

Leads to

You’re ready when you can

What interviewers check, from the resources below.

  • Compute accuracy, precision, recall and F1 from a 2x2 confusion matrix without notes
  • Explain why accuracy misleads on imbalanced data and name a case where recall beats precision (and the reverse)
  • Describe what moving the decision threshold does to precision and recall
  • State AUC as a probability about ranking a random positive above a random negative
  • Explain the difference between micro, macro and weighted averaging and when each hides a weak class
  • Write the F-beta formula and say what beta > 1 favours
  • Explain why log loss needs probabilities while F1 needs hard labels
  • Call precision_recall_fscore_support and roc_auc_score correctly on a binary problem
  • Read a reliability diagram and say whether a model is over- or under-confident
  • Explain why the calibrator must be fit on data the classifier never saw

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Classification: Accuracy, recall, precision, and related metrics (opens in a new tab)

    Google Machine Learning Crash Course · Google for Developers · 25 min · Free · checked 21 Oct 2026

    Why
    Interviewers expect you to compute precision and recall from a confusion matrix on the spot and to argue which one matters for a given product.
    What to read
    Read “Accuracy” through “(Optional, advanced) F1 score”, do “Exercise: Check your understanding”, then follow the Next link to “Classification: ROC and AUC” and read “Receiver-operating characteristic curve (ROC)” and “Area under the curve (AUC)”.
    What to skip
    The “Try it yourself” NaN box and the precision-recall-curve aside on the ROC page unless you have time.

  2. Free alternative

    3.4. Metrics and scoring: quantifying the quality of predictions (opens in a new tab)

    scikit-learn User Guide · scikit-learn developers · 30 min · Free · checked 21 Oct 2026

    Why
    The same metrics with the formulas, multiclass averaging rules and probability-based scores you will actually call in code when an interviewer asks you to evaluate a model.
    What to read
    Read “3.4.4.6. Confusion matrix”, “3.4.4.9. Precision, recall and F-measures” (both sub-sections), “3.4.4.12. Log loss” and “3.4.4.15. Receiver operating characteristic (ROC)” up to the binary case.
    What to skip
    Cohen's kappa, Hamming loss, hinge loss, Jaccard and the multilabel variants; the regression and clustering sections.

  3. Go deeper

    1.16. Probability calibration (opens in a new tab)

    scikit-learn User Guide · scikit-learn developers · 30 min · Free · checked 21 Oct 2026

    Why
    A follow-up question for stronger candidates: a model can rank well (high AUC) yet give probabilities you cannot act on, and LLM-based classifiers are notorious for this.
    What to read
    Read “1.16.1. Calibration curves”, “1.16.2. Calibrating a classifier”, then “1.16.3.1. Sigmoid”, “1.16.3.2. Isotonic” and “1.16.3.4. Temperature Scaling”.
    What to skip
    The CalibratedClassifierCV usage details and multiclass normalisation unless you plan to ship a calibrator.

Companies that test ML fundamentals most

Put Classification metrics in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan