Decision trees and tree ensembles
- ML fundamentals
- Asked often
- About 3 h to learn
- 3 vetted resources
What this topic covers
Decision trees split the data on one feature at a time, choosing each split to reduce impurity (Gini or entropy for classification, squared error for regression), and they overfit when grown deep. Random forests average many deep trees trained on bootstrap samples with random feature subsets, which cuts variance; gradient boosting adds shallow trees one at a time, each fitted to the current ensemble’s errors, which cuts bias. XGBoost and LightGBM make boosting fast and regularised, and they remain the default choice for tabular data. Interviewers check that you can explain how a split is chosen, contrast bagging with boosting in bias–variance terms, name the hyperparameters that control overfitting, and say why impurity-based feature importance can mislead.
You’re ready when you can
What interviewers check, from the resources below.
- Explain how a tree chooses a split for classification and for regression
- Explain why a single deep tree overfits and give two ways to stop it
- Describe bagging and attribute sampling in a random forest, and what out-of-bag evaluation gives you
- Describe what each new tree in gradient boosting is trained to predict and what shrinkage does
- Compare permutation importance with split-based importance
- Explain why random forests use bootstrap samples and random feature subsets, and which parameters matter most
- Describe gradient boosting as gradient descent in function space
- Explain how the learning rate and subsampling regularise boosting, and why a smaller rate needs more trees
- Name two reasons impurity-based importance can mislead and what to use instead
- Explain why binning features into histograms makes boosting much faster on large datasets
Decision trees and tree ensembles interview questions
Answer out loud, then open the points a strong answer covers. All Machine learning interview questions
How do random forests and gradient boosting differ, and when would you choose each?
- A random forest trains deep trees independently on bootstrap samples, testing a random subset of features at each split, and averaging their predictions mainly reduces variance.
- Gradient boosting adds shallow trees one after another, each fitted to the negative gradient of the loss for the current ensemble and scaled by a learning rate, which mainly reduces bias.
- Forests rarely overfit as trees are added and need little tuning, while boosting usually scores higher on tabular data but needs early stopping and tuning of learning rate, depth and tree count.
Your tree model ranks a customer ID column as its most important feature; what is going on, and how would you check?
- Impurity-based importance is computed on the training data and favours high-cardinality features, because a column with many unique values offers many splits that can fit noise.
- Compute permutation importance on a held-out set instead, which measures how much the validation score drops when that one feature’s values are shuffled.
- Correlated features share or swap credit under both methods, so group or drop them before concluding a feature matters, and treat importance as association with the prediction, not causation.
What to study, in order
Each link opened and checked by hand; the date says when.
Start here
Decision Forests (opens in a new tab)
Google for Developers · Google for Developers · 100 min · Free · checked 26 Oct 2026
- Why
- Tree ensembles win most tabular problems, so interviewers expect you to explain how a tree picks a split and how bagging and boosting each combine trees.
- What to read
- Read the decision tree “Overview” and “Conditions”, “Growing decision trees”, “Binary classification”, “Overfitting and pruning” and “Variable importances”; then “Random forests” and “Out-of-bag evaluation”; then “Gradient Boosted Decision Trees”, “The gradient boosting algorithm” and “Overfitting and regularization”. Do each “Check your understanding” page as you reach it.
- What to skip
- “Appropriate data”, the YDF code callouts, “A full example” and the random forest “Other topics” page.
Free alternative
1.11. Ensembles: Gradient boosting, random forests, bagging, voting, stacking (opens in a new tab)
scikit-learn User Guide · scikit-learn developers · 45 min · Free · checked 26 Oct 2026
- Why
- It states the boosting objective, how a forest randomises its trees and the known biases of impurity importance precisely, the level a follow-up question reaches.
- What to read
- Read “1.11.2.1. Random Forests”, “1.11.2.3. Parameters” and “1.11.2.5. Feature importance evaluation”; then the opening of “1.11.1.1. Histogram-Based Gradient Boosting”, “1.11.1.1.2. Missing values support” and “1.11.1.1.8. Why it’s faster”; then “1.11.1.2.3. Mathematical formulation” through “1.11.1.2.6. Subsampling”.
- What to skip
- The usage code, monotonic and interaction constraints, parallelism, extremely randomised trees, totally random trees embedding, voting, stacking and AdaBoost.
Go deeper
XGBoost: A Scalable Tree Boosting System (opens in a new tab)
arXiv · Tianqi Chen and Carlos Guestrin · 60 min · Free · checked 26 Oct 2026
- Why
- Interviewers who ask about XGBoost want its regularised objective and second-order split score, which the paper derives in about two pages.
- What to read
- Read “2. Tree Boosting in a Nutshell” (“2.1 Regularized Learning Objective”, “2.2 Gradient Tree Boosting”, “2.3 Shrinkage and Column Subsampling”), then “3.1 Basic Exact Greedy Algorithm”, “3.2 Approximate Algorithm” and “3.4 Sparsity-aware Split Finding”.
- What to skip
- The details of “3.3 Weighted Quantile Sketch”, “4. System Design”, the related work and the end-to-end evaluations.
Companies that test ML fundamentals most
Put Decision trees and tree ensembles in your plan
Tell us your interview date and hours a day; it lands on the right day, in the right order.