Deep learning basics
- ML fundamentals
- Asked often
- About 3 h to learn
- 3 vetted resources
What this topic covers
The working vocabulary of training neural networks beyond plain gradient descent: what convolutional and recurrent layers are built to exploit, and why self-attention replaced recurrence for sequences (parallel training and a short path between any two positions). It covers the optimisers you will be asked to compare (SGD, momentum, Adam), the learning-rate schedules that go with them (step decay, warmup, cosine), and the two layers that behave differently in training and inference, batch normalisation and dropout. ML Engineer interviewers use it to check that you can explain each update rule in a sentence or two, say what the extra state in Adam is for, justify a warmup, and predict what happens when a model is left in training mode at serving time.
Builds on
Leads to
You’re ready when you can
What interviewers check, from the resources below.
- Write the update rules for SGD, SGD with momentum and Adam, and say what each extra state variable stores
- Explain why the learning rate is decayed over training and name two common schedules
- Describe inverted dropout and why it leaves the network unchanged at test time
- State what batch normalisation normalises and why it makes training less sensitive to initialisation
- Explain how momentum smooths noisy minibatch gradients
- Derive Adam as momentum plus a per-parameter RMSProp scale, including why bias correction is needed early on
- Justify a learning-rate warmup and sketch a cosine schedule
- Compare CNNs, RNNs and self-attention on sequential operations and maximum path length
- Write the batch-norm transform, including the learned scale and shift
- Explain why inference uses population statistics instead of the current batch’s
Deep learning basics interview questions
Answer out loud, then open the points a strong answer covers. All Machine learning interview questions
How do SGD with momentum and Adam update weights, and why might you still choose SGD?
- SGD with momentum keeps a running average of past gradients and steps along it, which damps oscillation across steep directions and speeds progress along shallow ones.
- Adam also tracks a running average of squared gradients and divides each parameter’s step by its square root, giving every parameter its own effective learning rate, with bias correction early on.
- Adam usually converges faster with less tuning, but well-tuned SGD with momentum and a schedule can generalise as well or better on some vision tasks, and it stores less optimiser state.
How do batch normalisation and dropout behave differently in training and inference, and what goes wrong if you forget to switch modes?
- In training, batch norm normalises each feature with the current mini-batch’s mean and variance while updating running averages; at inference it uses those stored running averages instead.
- In training, inverted dropout zeroes each unit with probability p and scales the survivors by 1/(1 − p), so at inference it does nothing and expected activations still match.
- Serving in training mode makes predictions random and dependent on whatever else is in the batch, so call model.eval() before evaluation or serving and model.train() before resuming training.
What to study, in order
Each link opened and checked by hand; the date says when.
Start here
Neural Networks Part 3: Learning and Evaluation (CS231n notes) (opens in a new tab)
CS231n course notes · Stanford CS231n · 60 min · Free · checked 26 Oct 2026
- Why
- Comparing SGD, momentum and Adam, and saying why the learning rate is annealed, is a staple follow-up whenever an ML Engineer interview touches training.
- What to read
- Read “Parameter updates” in full: “SGD and bells and whistles” (vanilla update, momentum, Nesterov momentum), “Annealing the learning rate” and “Per-parameter adaptive learning rate methods” (Adagrad, RMSprop, Adam). Then open “Neural Networks Part 2: Setting up the Data and the Loss” from the CS231n notes index and read “Regularization” for dropout (vanilla and inverted) and the batch normalization paragraph under “Weight Initialization”.
- What to skip
- “Gradient Checks”, “Second order methods”, “Hyperparameter optimization” and “Model Ensembles”, plus Part 2’s data preprocessing and loss-function sections.
Free alternative
12. Optimization Algorithms (opens in a new tab)
Dive into Deep Learning · Aston Zhang, Zack C. Lipton, Mu Li and Alex J. Smola · 75 min · Free · checked 26 Oct 2026
- Why
- Interviewers often ask you to derive Adam from momentum and RMSProp and to justify a warmup, and this chapter builds each optimiser from the one before it.
- What to read
- Read 12.5 “Minibatch Stochastic Gradient Descent”, 12.6 “Momentum”, 12.8 “RMSProp”, 12.10 “Adam” (12.10.1 “The Algorithm”, including bias correction) and 12.11 “Learning Rate Scheduling” (12.11.3 “Policies”, especially “Cosine Scheduler” and “Warmup”). Then use the book’s sidebar to open 11.6 and read 11.6.2 “Comparing CNNs, RNNs, and Self-Attention”.
- What to skip
- 12.1–12.3 on convexity and full-batch gradient descent if you know them, 12.7 Adagrad, 12.9 Adadelta, 12.10.3 Yogi, and the implementation code unless you want to run it.
Go deeper
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (opens in a new tab)
arXiv · Sergey Ioffe and Christian Szegedy · 60 min · Free · checked 26 Oct 2026
- Why
- Batch norm’s different behaviour in training and inference is a classic source of production bugs, and interviewers like to hear it explained from the original paper.
- What to read
- Read the abstract, section 1 “Introduction”, section 2 “Towards Reducing Internal Covariate Shift” and section 3 “Normalization via Mini-Batch Statistics” through 3.1 “Training and Inference with Batch-Normalized Networks”, with Algorithm 1 (the transform) and Algorithm 2 (training and inference). Skim 3.3 “Batch Normalization enables higher learning rates”.
- What to skip
- 3.2 on convolutional networks, 3.4 on regularisation unless you have time, and section 4 “Experiments”.
Companies that test ML fundamentals most
Put Deep learning basics in your plan
Tell us your interview date and hours a day; it lands on the right day, in the right order.