Monitoring ML models and data drift
- ML in production
- Asked often
- About 2 h 50 min to learn
- 3 vetted resources
What this topic covers
Models usually fail quietly: the code runs, predictions come back, and quality decays because the data has moved. This topic separates covariate shift, label shift and concept drift from training–serving skew, which is a mismatch between the two pipelines rather than a change in the world. Since labels often arrive late or never, you monitor what you can see: input feature distributions, the prediction distribution, data quality and proxy outcomes such as clicks, each compared against a reference window with a statistical test or a distance. Interviewers check that you can pick the right signals for a given product, explain why a drift alert is a reason to investigate rather than an automatic retraining trigger, and choose between retraining on a schedule and retraining on evidence.
Builds on
You’re ready when you can
What interviewers check, from the resources below.
- Define covariate shift, label shift and concept drift with one example each
- Explain natural labels and feedback-loop length and how they limit what you can monitor
- Pick a two-sample test for a feature and explain why the reference window matters
- List the ML-specific signals to monitor, from accuracy-related metrics down to raw inputs, and say which need labels
- Explain a degenerate feedback loop in a recommender and one way to break it
- Separate data drift, concept drift, prediction drift and training–serving skew in one sentence each
- Name a statistical test and a distance metric for detecting drift and say when you would pick each
- Explain why input and prediction drift serve as proxies when labels are delayed
- Argue why drift alone should not trigger automatic retraining and list the other ways to respond
- Frame shift detection as a two-sample test between a reference sample and recent production data
Monitoring ML models and data drift interview questions
Answer out loud, then open the points a strong answer covers. All Machine learning interview questions
What is the difference between covariate shift, label shift and concept drift?
- Covariate shift means the input distribution P(X) changes while P(Y|X) stays the same, for example when a marketing campaign brings in users from a new age group.
- Label shift means the output distribution P(Y) changes while P(X|Y) stays the same, for example when a disease becomes more common but its symptoms look exactly as before.
- Concept drift means P(Y|X) itself changes, so the same input now deserves a different answer, as when buyer behaviour changes after a price rise; only labels or good proxies reveal it reliably.
Labels for your model arrive weeks late. How would you monitor it in the meantime?
- Compare the distributions of key input features and of the predictions against a reference window from training or a recent healthy period, using a test such as Kolmogorov–Smirnov or a distance such as PSI.
- Add data quality checks for schema, missing values and out-of-range values, because many apparent quality drops come from broken upstream pipelines rather than a changing world.
- Treat a drift alert as a prompt to investigate rather than an automatic retraining trigger, and confirm real impact with early proxy outcomes, such as clicks or complaints, before acting.
What to study, in order
Each link opened and checked by hand; the date says when.
Start here
Data Distribution Shifts and Monitoring (opens in a new tab)
huyenchip.com · Chip Huyen · 75 min · Free · checked 26 Oct 2026
- Why
- It covers the whole answer to “your model got worse in production”: why ML systems fail, the kinds of shift, how to detect them, and what to monitor while labels are slow to arrive.
- What to read
- Read “Natural Labels and Feedback Loop”, “Causes of ML System Failures” (skim “Edge Cases”), all of “Data Distribution Shifts” (the types of shift, “Detecting Data Distribution Shifts” with “Statistical methods” and “Time scale windows for detecting shifts”, then “Addressing Data Distribution Shifts”), and “ML-Specific Metrics” under “Monitoring and Observability”.
- What to skip
- “Monitoring Toolbox” (logs, dashboards, alerts) and “Observability”, which are general software engineering, and the footnotes.
Free alternative
What is data drift in ML, and how to detect and handle it (opens in a new tab)
Evidently AI · Evidently AI · 35 min · Free · checked 26 Oct 2026
- Why
- Interviewers test whether you can tell data drift from concept drift, prediction drift and training–serving skew, and this guide draws each of those lines before it lists detection methods.
- What to read
- Read “What is data drift?” with all five comparisons beneath it (concept drift, prediction drift, training-serving skew, data quality, outlier detection), “Why is data drift important?”, “How to detect data drift” (summary statistics, statistical tests, distance metrics, rule-based checks) and “How to handle data drift”.
- What to skip
- “Detecting drift with Evidently” and the sign-up prompts, which promote the product, and “Additional resources”.
Go deeper
Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift (opens in a new tab)
arXiv (NeurIPS 2019) · Stephan Rabanser, Stephan Günnemann, Zachary C. Lipton · 60 min · Free · checked 26 Oct 2026
- Why
- When labels are delayed, detecting shift from inputs alone is the core monitoring problem, and this paper tests the practical recipe a strong answer describes: reduce dimensions, run two-sample tests, and correct for running many of them.
- What to read
- Read “1 Introduction”, all of section 3 (“3.1 Dimensionality Reduction”, “3.2 Statistical Hypothesis Testing”, “3.3 Obtaining Most Anomalous Samples” and “3.4 Determining the Malignancy of a Shift”), then “5 Discussion” for the findings and “6 Conclusions”.
- What to skip
- “2 Related work”, the dataset and shift set-up details in “4 Experiments”, and the appendix tables and figures.
Put Monitoring ML models and data drift in your plan
Tell us your interview date and hours a day; it lands on the right day, in the right order.