A neural network from scratch in NumPy
- ML fundamentals
- Specialist topic or differentiator
- About 75 min to learn
- 3 vetted resources
What this topic covers
Build a tiny multilayer perceptron with nothing but NumPy: a forward pass of matrix multiplies and nonlinearities, a loss, hand-derived gradients via the chain rule, and a gradient-descent update loop. Interviewers use this as the fundamentals coding round for applied roles: you are handed a whiteboard or a blank notebook and asked to write forward and backward passes for a two-layer net, then explain why each gradient looks the way it does. Expect follow-ups on vectorising over a batch, why initialisation and learning rate matter, and how this maps onto what PyTorch's autograd does for you. Being able to do it cold also makes every later topic (attention, LoRA, KV cache) feel like plumbing rather than magic.
Builds on
You’re ready when you can
What interviewers check, from the resources below.
- Implement a 2-2-1 sigmoid network forward pass in NumPy without looking
- Derive the partial derivative of MSE loss with respect to one hidden-layer weight using the chain rule
- Explain why the sigmoid derivative can be written in terms of the sigmoid output
- Rewrite the one-sample-at-a-time SGD loop to process a whole batch with matrix operations
- Implement a neuron's forward pass, loss and weight update in a single function
- State the gradient of the loss with respect to weights and bias in one line each
- Extend the single neuron to a two-layer network and explain which extra gradient terms appear
- Explain how a computation graph is recorded during the forward pass and walked in reverse during backward
- Implement the backward rule for one new operation (e.g. tanh or division) in the Value class
- Compare scalar-valued autograd with the tensor-valued version in PyTorch and say where the speed comes from
What to study, in order
Each link opened and checked by hand; the date says when.
Start here
Machine Learning for Beginners: An Introduction to Neural Networks (opens in a new tab)
victorzhou.com · Victor Zhou · 45 min · Free · checked 21 Oct 2026
- Why
- The classic fundamentals round is “write a two-layer net and its backward pass in NumPy”, and this walkthrough derives every gradient you would have to write on the board.
- What to read
- Read “Building Blocks: Neurons” through “Training a Neural Network, Part 2”, and type out “Code: A Complete Neural Network” yourself rather than copying it.
- What to skip
- “Now What?” and the suggested follow-up series; the toy height/weight dataset itself is irrelevant, the derivation is the point.
Free alternative
Single Neuron with Backpropagation (opens in a new tab)
Deep-ML · Deep-ML · 30 min · Free · checked 21 Oct 2026
- Why
- A timed, test-graded version of exactly what an interviewer asks: forward pass, gradient, update loop, no framework.
- What to read
- Solve the whole problem in NumPy against the hidden tests, then extend your own solution to a hidden layer so it becomes a full MLP.
- What to skip
- Don't reach for the site's hint or solution until you have a passing or clearly stuck attempt; the value is in writing the gradient yourself.
Go deeper
micrograd: A tiny scalar-valued autograd engine and a neural net library on top of it with PyTorch-like API (opens in a new tab)
GitHub · Andrej Karpathy · 60 min · Free · checked 21 Oct 2026
- Why
- Once you can hand-write backprop, the next interview question is “so what does autograd actually do?”, and this answers it in roughly 150 lines.
- What to read
- Read the README sections “Example usage”, “Training a neural net” and “Tracing / visualization”, then read micrograd/engine.py and micrograd/nn.py in full and run demo.ipynb.
- What to skip
- “Training a GPT” and the microgpt follow-up; setup.py and the test folder.
Companies that test ML fundamentals most
Put A neural network from scratch in NumPy in your plan
Tell us your interview date and hours a day; it lands on the right day, in the right order.