Gradient descent and backpropagation
- ML fundamentals
- Asked often
- About 1 h 50 min to learn
- 3 vetted resources
What this topic covers
Gradient descent is the loop every model is trained with: compute a loss, get its gradient with respect to the parameters, step the parameters a little in the opposite direction, repeat. Backpropagation is how that gradient is computed efficiently for a composed function such as a network, by applying the chain rule backwards through the computation graph one operation at a time. Interviewers use this topic as a calibration check for Applied AI roles: explain the update rule and the role of the learning rate, derive the gradient through a small expression by hand, say what each gate does to the gradient flowing through it, and write a short training loop in plain Python or NumPy. Follow-ups usually probe mini-batch vs full-batch, why gradients vanish, and how an autograd engine like PyTorch's actually works.
You’re ready when you can
What interviewers check, from the resources below.
- Explain what a gradient tells you about a small change in each input
- Derive the backward pass for a two- or three-gate expression by hand using local gradient times upstream gradient
- State how add, max and multiply gates each route the gradient
- Explain why a sigmoid saturating kills the gradient flowing through it
- Write the mean-squared-error gradient for a linear model in vectorised NumPy
- Implement the update loop with a learning rate and a fixed number of iterations
- Explain what happens when the learning rate is too large or too small
- Explain the difference between this full-batch update and a mini-batch or stochastic one
- Explain how a computation graph is recorded during the forward pass and walked in reverse during backward
- Implement the backward rule for one new operation (e.g. tanh or division) in the Value class
What to study, in order
Each link opened and checked by hand; the date says when.
Start here
Backpropagation, Intuitions (CS231n notes) (opens in a new tab)
CS231n course notes · Stanford CS231n · 35 min · Free · checked 21 Oct 2026
- Why
- The standard whiteboard question is “walk the gradient back through this expression”, and these notes teach exactly the gate-by-gate way of thinking that makes it mechanical.
- What to read
- Read “Introduction” through “Patterns in backward flow”; work the staged-computation example on paper before reading its answer.
- What to skip
- “Gradients for vectorized operations” unless you have time; it matters for implementing layers, not for the usual interview derivation.
Free alternative
Linear Regression Using Gradient Descent (opens in a new tab)
Deep-ML · Deep-ML · 30 min · Free · checked 21 Oct 2026
- Why
- “Implement gradient descent for a linear model in NumPy” is one of the most common 20-minute coding screens for applied ML roles.
- What to read
- Solve the problem in the in-browser editor without looking at the solution: write the prediction, the loss gradient and the update loop yourself, then run the tests.
- What to skip
- Nothing; it is a single problem. Do not peek at the solution until your own version passes or you are properly stuck.
Go deeper
micrograd: A tiny scalar-valued autograd engine and a neural net library on top of it with PyTorch-like API (opens in a new tab)
GitHub · Andrej Karpathy · 60 min · Free · checked 21 Oct 2026
- Why
- Once you can hand-write backprop, the next interview question is “so what does autograd actually do?”, and this answers it in roughly 150 lines.
- What to read
- Read the README sections “Example usage”, “Training a neural net” and “Tracing / visualization”, then read micrograd/engine.py and micrograd/nn.py in full and run demo.ipynb.
- What to skip
- “Training a GPT” and the microgpt follow-up; setup.py and the test folder.
Companies that test ML fundamentals most
Put Gradient descent and backpropagation in your plan
Tell us your interview date and hours a day; it lands on the right day, in the right order.