# ML pipelines, feature stores and reproducibility > What interviewers check on ML pipelines, feature stores and reproducibility, and what to study, in order. Canonical: https://saadho.arushk.tech/topics/ml-pipelines-feature-stores Last updated: 2026-10-26 ML in production. Asked often. About 2 h 10 min to learn. 3 vetted resources. ## What this topic covers What sits around the model: data pipelines that build features, training pipelines that retrain and validate on a schedule or a trigger, and the metadata that records which data, code and settings produced each model. A feature store keeps one definition per feature, serves it from an offline store for training and a low-latency online store for inference, and joins features to labels as of each example’s timestamp so no future data leaks in. Interviewers check that you can explain point-in-time correctness with an example, say how a feature store reduces training–serving skew, describe what an experiment tracker records, and make a training run reproducible months later. Knowing the classic sources of ML technical debt, such as pipeline jungles and entangled features, helps in senior rounds. ### Builds on - [Deploying and serving ML models](https://saadho.arushk.tech/topics/ml-deployment-serving) ## You’re ready when you can - Describe MLOps levels 0, 1 and 2 and what changes at each - Explain continuous training and name three triggers for a pipeline run - Say what data validation and model validation each check before a model is promoted - List what a metadata store records so any model can be traced and reproduced - Explain offline versus online feature stores and how values get from one to the other - Describe point-in-time correctness with an example of the leak it prevents - Say how one shared feature definition reduces training–serving skew - Argue when a team does not need a feature store yet - Explain “changing anything changes everything” with an example of entangled features - Describe a pipeline jungle and how a shared feature pipeline or feature store prevents it ## ML pipelines, feature stores and reproducibility interview questions **What is point-in-time correctness in a feature store, and what goes wrong without it?** - Each training example must use feature values as they were known at that example’s timestamp, so labels are joined to features with an as-of join on time, not to the latest values. - Joining on the latest values leaks future information into training, so offline metrics look strong while the model disappoints in production, where only past data is available. - A feature store keeps timestamped history in an offline store for training and the latest values in a low-latency online store, both computed from one definition to avoid skew. **How would you make a training run reproducible six months from now?** - Version the code, the configuration and the exact data snapshot or query with its time range, since rerunning a query against a changed table silently trains on different data. - Pin the environment with locked dependency versions or a container image and fix random seeds, accepting that GPU nondeterminism can still cause small differences between runs. - Log each run’s parameters, data version, metrics and model artefact in an experiment tracker or metadata store, so every served model can be traced back to what produced it. ## What to study, in order ### 1. MLOps: Continuous delivery and automation pipelines in machine learning Start here. Google Cloud Architecture Center, Google Cloud. Guide, about 45 min. Free. Last checked 2026-10-26. [Open](https://docs.cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning) - **Why:** Its three maturity levels give a ready structure for “how would you automate retraining?”, and it puts the feature store, metadata store and pipeline triggers exactly where interviewers expect them. - **Read:** Read “DevOps versus MLOps”, “Data science steps for ML”, “MLOps level 0: Manual process” (characteristics and challenges), all of “MLOps level 1: ML pipeline automation” including “Data and model validation”, “Feature store”, “Metadata management” and “ML pipeline triggers”, then “MLOps level 2: CI/CD pipeline automation”. - **Skip:** “What’s next”, the contributors list and the links out to Vertex AI product pages. ### 2. Feature Stores: A Hierarchy of Needs Free alternative. eugeneyan.com, Eugene Yan. Article, about 40 min. Free. Last checked 2026-10-26. [Open](https://eugeneyan.com/writing/feature-stores/) - **Why:** Interviewers ask what a feature store is actually for, and this post answers from the needs upwards (reuse, serving, integrity, convenience, automation) with examples from Uber, Netflix, Airbnb, GoJek and others. - **Read:** Read “The feature store hierarchy of needs”, then “Access”, “Serving” and “Integrity” closely (offline and online stores, train-serve skew, and Netflix’s time travel for point-in-time correctness), and skim “Convenience”, “Autopilot” (backfills) and “Conclusion: It depends on what you need”. - **Skip:** The references list and the names of each company’s internal tools. ### 3. Hidden Technical Debt in Machine Learning Systems Go deeper. NeurIPS 2015 (Google), D. Sculley et al.. Paper, about 45 min. Free. Last checked 2026-10-26. [Open](https://papers.nips.cc/paper_files/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html) - **Why:** It is the standard reference for why ML systems cost so much to maintain, and its terms (entanglement, pipeline jungles, undeclared consumers, configuration debt) come up in senior design and production rounds. - **Read:** Open the paper PDF from this page and read “1 Introduction”, “2 Complex Models Erode Boundaries”, “3 Data Dependencies Cost More than Code Dependencies”, “4 Feedback Loops”, “5 ML-System Anti-Patterns” (glue code, pipeline jungles, dead experimental codepaths), “6 Configuration Debt”, “7 Dealing with Changes in the External World” and the short “9 Conclusions: Measuring Debt and Paying it Off”. - **Skip:** “8 Other Areas of ML-related Debt” except its paragraph on reproducibility debt. ## Related - [All topics](https://saadho.arushk.tech/library) - [Build a plan that includes this](https://saadho.arushk.tech/start)