# Monitoring ML models and data drift

> What interviewers check on Monitoring ML models and data drift, and what to study, in order.

Canonical: https://saadho.arushk.tech/topics/ml-monitoring-drift
Last updated: 2026-10-26

ML in production. Asked often. About 2 h 50 min to learn. 3 vetted resources.

## What this topic covers

Models usually fail quietly: the code runs, predictions come back, and quality decays because the data has moved. This topic separates covariate shift, label shift and concept drift from training–serving skew, which is a mismatch between the two pipelines rather than a change in the world. Since labels often arrive late or never, you monitor what you can see: input feature distributions, the prediction distribution, data quality and proxy outcomes such as clicks, each compared against a reference window with a statistical test or a distance. Interviewers check that you can pick the right signals for a given product, explain why a drift alert is a reason to investigate rather than an automatic retraining trigger, and choose between retraining on a schedule and retraining on evidence.

### Builds on

- [Deploying and serving ML models](https://saadho.arushk.tech/topics/ml-deployment-serving)

## You’re ready when you can

- Define covariate shift, label shift and concept drift with one example each
- Explain natural labels and feedback-loop length and how they limit what you can monitor
- Pick a two-sample test for a feature and explain why the reference window matters
- List the ML-specific signals to monitor, from accuracy-related metrics down to raw inputs, and say which need labels
- Explain a degenerate feedback loop in a recommender and one way to break it
- Separate data drift, concept drift, prediction drift and training–serving skew in one sentence each
- Name a statistical test and a distance metric for detecting drift and say when you would pick each
- Explain why input and prediction drift serve as proxies when labels are delayed
- Argue why drift alone should not trigger automatic retraining and list the other ways to respond
- Frame shift detection as a two-sample test between a reference sample and recent production data

## Monitoring ML models and data drift interview questions

**What is the difference between covariate shift, label shift and concept drift?**

- Covariate shift means the input distribution P(X) changes while P(Y|X) stays the same, for example when a marketing campaign brings in users from a new age group.
- Label shift means the output distribution P(Y) changes while P(X|Y) stays the same, for example when a disease becomes more common but its symptoms look exactly as before.
- Concept drift means P(Y|X) itself changes, so the same input now deserves a different answer, as when buyer behaviour changes after a price rise; only labels or good proxies reveal it reliably.

**Labels for your model arrive weeks late. How would you monitor it in the meantime?**

- Compare the distributions of key input features and of the predictions against a reference window from training or a recent healthy period, using a test such as Kolmogorov–Smirnov or a distance such as PSI.
- Add data quality checks for schema, missing values and out-of-range values, because many apparent quality drops come from broken upstream pipelines rather than a changing world.
- Treat a drift alert as a prompt to investigate rather than an automatic retraining trigger, and confirm real impact with early proxy outcomes, such as clicks or complaints, before acting.

## What to study, in order

### 1. Data Distribution Shifts and Monitoring

Start here. huyenchip.com, Chip Huyen. Article, about 75 min. Free. Last checked 2026-10-26. [Open](https://huyenchip.com/2022/02/07/data-distribution-shifts-and-monitoring.html)

- **Why:** It covers the whole answer to “your model got worse in production”: why ML systems fail, the kinds of shift, how to detect them, and what to monitor while labels are slow to arrive.
- **Read:** Read “Natural Labels and Feedback Loop”, “Causes of ML System Failures” (skim “Edge Cases”), all of “Data Distribution Shifts” (the types of shift, “Detecting Data Distribution Shifts” with “Statistical methods” and “Time scale windows for detecting shifts”, then “Addressing Data Distribution Shifts”), and “ML-Specific Metrics” under “Monitoring and Observability”.
- **Skip:** “Monitoring Toolbox” (logs, dashboards, alerts) and “Observability”, which are general software engineering, and the footnotes.

### 2. What is data drift in ML, and how to detect and handle it

Free alternative. Evidently AI, Evidently AI. Guide, about 35 min. Free. Last checked 2026-10-26. [Open](https://www.evidentlyai.com/ml-in-production/data-drift)

- **Why:** Interviewers test whether you can tell data drift from concept drift, prediction drift and training–serving skew, and this guide draws each of those lines before it lists detection methods.
- **Read:** Read “What is data drift?” with all five comparisons beneath it (concept drift, prediction drift, training-serving skew, data quality, outlier detection), “Why is data drift important?”, “How to detect data drift” (summary statistics, statistical tests, distance metrics, rule-based checks) and “How to handle data drift”.
- **Skip:** “Detecting drift with Evidently” and the sign-up prompts, which promote the product, and “Additional resources”.

### 3. Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift

Go deeper. arXiv (NeurIPS 2019), Stephan Rabanser, Stephan Günnemann, Zachary C. Lipton. Paper, about 60 min. Free. Last checked 2026-10-26. [Open](https://arxiv.org/abs/1810.11953)

- **Why:** When labels are delayed, detecting shift from inputs alone is the core monitoring problem, and this paper tests the practical recipe a strong answer describes: reduce dimensions, run two-sample tests, and correct for running many of them.
- **Read:** Read “1 Introduction”, all of section 3 (“3.1 Dimensionality Reduction”, “3.2 Statistical Hypothesis Testing”, “3.3 Obtaining Most Anomalous Samples” and “3.4 Determining the Malignancy of a Shift”), then “5 Discussion” for the findings and “6 Conclusions”.
- **Skip:** “2 Related work”, the dataset and shift set-up details in “4 Experiments”, and the appendix tables and figures.

## Related

- [All topics](https://saadho.arushk.tech/library)
- [Build a plan that includes this](https://saadho.arushk.tech/start)