Deploying and serving ML models
- ML in production
- Asked in most interview loops
- About 2 h 25 min to learn
- 3 vetted resources
What this topic covers
Getting a trained model in front of users and keeping it there. The first decision is batch inference, where predictions are precomputed and looked up, versus online inference, where each request is scored live inside a latency budget; many systems mix the two. Then come packaging the model with its preprocessing so training and serving agree, choosing between embedding it in an application, running it as a separate service or publishing it as data, and validating a new version before it takes traffic. Interviewers check that you can justify the batch or online choice, split a latency budget across feature lookup and inference, and describe a release plan with shadow traffic, a canary, the metrics you watch and a fast rollback.
You’re ready when you can
What interviewers check, from the resources below.
- Compare static and dynamic inference on cost, latency, coverage of rare inputs and the chance to check predictions first
- Pick batch or online inference for a given product and defend the choice
- Explain why transforming data inside the model, rather than in a separate script, keeps training and serving consistent
- List the checks a new model must pass before it takes traffic, including comparison with the version it replaces
- Name the embedded model, model-as-a-service and model-as-data patterns and give a case for each
- Explain shadow deployment and what you compare before promoting the shadow model
- Describe a canary or competing-models rollout and the metrics that would stop it
- Say what must already be in place for a rollback to take minutes rather than a retrain
- Describe the three stages from batch prediction to online prediction with streaming features and when each is enough
- Split a request’s latency between feature computation, feature lookup and model inference
Deploying and serving ML models interview questions
Answer out loud, then open the points a strong answer covers. All Machine learning interview questions
When would you serve predictions in batch rather than online?
- Batch inference precomputes predictions on a schedule and stores them for lookup, which suits a bounded set of inputs such as known users or items and tolerates somewhat stale results.
- Online inference scores each request as it arrives, so it handles new or free-form inputs and fresh context, but it must fit a latency budget and needs heavier serving and monitoring.
- A common hybrid precomputes heavy features or candidates in batch and runs a lighter model online, keeping most of the cost offline while still reacting to the current request.
How would you roll out a new model version safely?
- Before any traffic, compare the candidate offline against the current model and a fixed quality bar, and check that it loads and meets latency targets on the real serving stack.
- Run it in shadow mode first, scoring live requests without affecting users, so you can compare its predictions, latency and error rate with production at no risk to users.
- Then send a small canary share of traffic, watch business and system metrics against the old version, and keep the previous model deployed so rollback is a quick configuration switch.
What to study, in order
Each link opened and checked by hand; the date says when.
Start here
Production ML systems (opens in a new tab)
Google Machine Learning Crash Course · Google for Developers · 50 min · Free · checked 26 Oct 2026
- Why
- Batch versus online inference is usually the first serving decision an interviewer asks you to defend, and this module sets out its trade-offs in cost, latency, coverage of rare inputs and checking predictions before they ship.
- What to read
- Read “Introduction”, “Static vs. dynamic training”, “Static vs. dynamic inference”, “When to transform data?” and “Deployment testing” (especially “Validate model quality before serving” and “Validate model-infrastructure compatibility before serving”), then do “Test your knowledge”.
- What to skip
- “Monitoring pipelines” and “Questions to ask” for now; come back to “Monitoring pipelines” when you study ml-monitoring-drift.
Free alternative
Continuous Delivery for Machine Learning (opens in a new tab)
martinfowler.com · Danilo Sato, Arif Wider, Christoph Windheuser · 45 min · Free · checked 26 Oct 2026
- Why
- It names the three ways to package a model for serving and the release patterns (shadow models, competing models, canaries, rollback) that interviewers expect in any deployment answer.
- What to read
- Read “Introduction and Definition”, then under “Technical Components of CD4ML” read “Model Serving”, “Testing and Quality in Machine Learning”, “Model Deployment”, “Continuous Delivery Orchestration” (with “Deployment Pipelines”) and “Model Monitoring and Observability”.
- What to skip
- “A Machine Learning Application for Sales Forecasting”, the data, training and experiment-tracking sections (they belong to ml-pipelines-feature-stores) and “Where do We Go From Here?”.
Go deeper
Real-time machine learning: challenges and solutions (opens in a new tab)
huyenchip.com · Chip Huyen · 50 min · Free · checked 26 Oct 2026
- Why
- Senior interviewers push past “just serve it online” to ask where the latency goes and how fresh features reach the model, and this post walks through each step from batch prediction to streaming features.
- What to read
- Read “Towards Online Prediction” in full: “Stage 1. Batch prediction”, “Stage 2. Online prediction with batch features” and “Stage 3. Online prediction with online features”, plus the appendix note “What do feature stores do?”.
- What to skip
- The bandits discussion and the whole “Towards Continual Learning” half unless you have time; skim the appendix on stream versus batch efficiency.
Put Deploying and serving ML models in your plan
Tell us your interview date and hours a day; it lands on the right day, in the right order.