# Decision trees and tree ensembles

> What interviewers check on Decision trees and tree ensembles, and what to study, in order.

Canonical: https://saadho.arushk.tech/topics/tree-ensembles
Last updated: 2026-10-26

ML fundamentals. Asked often. About 3 h to learn. 3 vetted resources.

## What this topic covers

Decision trees split the data on one feature at a time, choosing each split to reduce impurity (Gini or entropy for classification, squared error for regression), and they overfit when grown deep. Random forests average many deep trees trained on bootstrap samples with random feature subsets, which cuts variance; gradient boosting adds shallow trees one at a time, each fitted to the current ensemble’s errors, which cuts bias. XGBoost and LightGBM make boosting fast and regularised, and they remain the default choice for tabular data. Interviewers check that you can explain how a split is chosen, contrast bagging with boosting in bias–variance terms, name the hyperparameters that control overfitting, and say why impurity-based feature importance can mislead.

### Builds on

- [Bias/variance, overfitting and regularisation](https://saadho.arushk.tech/topics/bias-variance-regularisation)

## You’re ready when you can

- Explain how a tree chooses a split for classification and for regression
- Explain why a single deep tree overfits and give two ways to stop it
- Describe bagging and attribute sampling in a random forest, and what out-of-bag evaluation gives you
- Describe what each new tree in gradient boosting is trained to predict and what shrinkage does
- Compare permutation importance with split-based importance
- Explain why random forests use bootstrap samples and random feature subsets, and which parameters matter most
- Describe gradient boosting as gradient descent in function space
- Explain how the learning rate and subsampling regularise boosting, and why a smaller rate needs more trees
- Name two reasons impurity-based importance can mislead and what to use instead
- Explain why binning features into histograms makes boosting much faster on large datasets

## Decision trees and tree ensembles interview questions

**How do random forests and gradient boosting differ, and when would you choose each?**

- A random forest trains deep trees independently on bootstrap samples, testing a random subset of features at each split, and averaging their predictions mainly reduces variance.
- Gradient boosting adds shallow trees one after another, each fitted to the negative gradient of the loss for the current ensemble and scaled by a learning rate, which mainly reduces bias.
- Forests rarely overfit as trees are added and need little tuning, while boosting usually scores higher on tabular data but needs early stopping and tuning of learning rate, depth and tree count.

**Your tree model ranks a customer ID column as its most important feature; what is going on, and how would you check?**

- Impurity-based importance is computed on the training data and favours high-cardinality features, because a column with many unique values offers many splits that can fit noise.
- Compute permutation importance on a held-out set instead, which measures how much the validation score drops when that one feature’s values are shuffled.
- Correlated features share or swap credit under both methods, so group or drop them before concluding a feature matters, and treat importance as association with the prediction, not causation.

## What to study, in order

### 1. Decision Forests

Start here. Google for Developers, Google for Developers. Course, about 1 h 40 min. Free. Last checked 2026-10-26. [Open](https://developers.google.com/machine-learning/decision-forests)

- **Why:** Tree ensembles win most tabular problems, so interviewers expect you to explain how a tree picks a split and how bagging and boosting each combine trees.
- **Read:** Read the decision tree “Overview” and “Conditions”, “Growing decision trees”, “Binary classification”, “Overfitting and pruning” and “Variable importances”; then “Random forests” and “Out-of-bag evaluation”; then “Gradient Boosted Decision Trees”, “The gradient boosting algorithm” and “Overfitting and regularization”. Do each “Check your understanding” page as you reach it.
- **Skip:** “Appropriate data”, the YDF code callouts, “A full example” and the random forest “Other topics” page.

### 2. 1.11. Ensembles: Gradient boosting, random forests, bagging, voting, stacking

Free alternative. scikit-learn User Guide, scikit-learn developers. Guide, about 45 min. Free. Last checked 2026-10-26. [Open](https://scikit-learn.org/stable/modules/ensemble.html)

- **Why:** It states the boosting objective, how a forest randomises its trees and the known biases of impurity importance precisely, the level a follow-up question reaches.
- **Read:** Read “1.11.2.1. Random Forests”, “1.11.2.3. Parameters” and “1.11.2.5. Feature importance evaluation”; then the opening of “1.11.1.1. Histogram-Based Gradient Boosting”, “1.11.1.1.2. Missing values support” and “1.11.1.1.8. Why it’s faster”; then “1.11.1.2.3. Mathematical formulation” through “1.11.1.2.6. Subsampling”.
- **Skip:** The usage code, monotonic and interaction constraints, parallelism, extremely randomised trees, totally random trees embedding, voting, stacking and AdaBoost.

### 3. XGBoost: A Scalable Tree Boosting System

Go deeper. arXiv, Tianqi Chen and Carlos Guestrin. Paper, about 60 min. Free. Last checked 2026-10-26. [Open](https://arxiv.org/abs/1603.02754)

- **Why:** Interviewers who ask about XGBoost want its regularised objective and second-order split score, which the paper derives in about two pages.
- **Read:** Read “2. Tree Boosting in a Nutshell” (“2.1 Regularized Learning Objective”, “2.2 Gradient Tree Boosting”, “2.3 Shrinkage and Column Subsampling”), then “3.1 Basic Exact Greedy Algorithm”, “3.2 Approximate Algorithm” and “3.4 Sparsity-aware Split Finding”.
- **Skip:** The details of “3.3 Weighted Quantile Sketch”, “4. System Design”, the related work and the end-to-end evaluations.

## Companies that test this area most

- [Google DeepMind](https://saadho.arushk.tech/companies/google-deepmind)
- [Flipkart](https://saadho.arushk.tech/companies/flipkart)
- [Sarvam AI](https://saadho.arushk.tech/companies/sarvam-ai)
- [Krutrim](https://saadho.arushk.tech/companies/krutrim)
- [Info Edge (Naukri)](https://saadho.arushk.tech/companies/info-edge)
- [Amazon](https://saadho.arushk.tech/companies/amazon)

## Related

- [All topics](https://saadho.arushk.tech/library)
- [Build a plan that includes this](https://saadho.arushk.tech/start)