Skip to content

Self-attention from scratch

  • Transformers
  • Asked in most interview loops
  • About 2 h to learn
  • 4 vetted resources
Build a plan that includes this

What this topic covers

Self-attention lets every token look at every other token and decide what matters. Each token becomes a query, a key and a value; scaled dot products between queries and keys become weights over the values. Interviews ask you to write it in NumPy or PyTorch, including the causal mask and multiple heads, and to explain why the scores are divided by √d.

Leads to

You’re ready when you can

What interviewers check, from the resources below.

  • Correct output shape for (seq_len, d) inputs
  • Numerically stable softmax (subtract the row max)
  • Scaled by √d_k
  • Can explain the O(n²) memory cost out loud
  • Build a causal mask with torch.triu and apply it before softmax
  • Split and merge heads with view/transpose without shape bugs
  • Explain where dropout goes in attention
  • Explain what queries, keys and values are, in one sentence each
  • Draw the Q·Kᵀ → softmax → ×V pipeline with tensor shapes
  • Say why scores are divided by √d before the softmax

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Implement Self-Attention Mechanism (opens in a new tab)

    Deep-ML · 30 min · Free · checked 20 Oct 2026

    Why
    “Implement attention from scratch” is one of the most common ML coding prompts. This is the exact drill, with tests.
    What to read
    Solve it in NumPy in under 30 minutes without looking at the solution. Then try #107 (masked self-attention).
    What to skip
    Don't optimise; correct shapes and a clean softmax are what's graded.

  2. Free alternative

    LLMs-from-scratch, Chapter 3: Coding attention mechanisms (opens in a new tab)

    GitHub · rasbt/LLMs-from-scratch · Sebastian Raschka · 60 min · Free · checked 20 Oct 2026

    Why
    Production-quality PyTorch for causal and multi-head attention: the version you'd want to write in a PyTorch round.
    What to read
    Work through the notebook in 01_main-chapter-code; re-type the causal attention class from memory afterwards.
    What to skip
    The bonus folders (efficient multi-head attention, buffers) until you've done the main notebook.

  3. Go deeper

    The Illustrated Transformer (opens in a new tab)

    jalammar.github.io · Jay Alammar · 30 min · Free · checked 20 Oct 2026

    Why
    The clearest visual model of attention. Interviewers expect you to draw this on a whiteboard, not just code it.
    What to read
    Read “Self-Attention at a High Level” through “The Beast With Many Heads”, then “Positional Encoding”.
    What to skip
    The decoder, loss function and training recap: not needed for an attention question.

  4. Go deeper

    Attention Is All You Need (opens in a new tab)

    arXiv · Ashish Vaswani et al. · 60 min · Free · checked 22 Oct 2026

    Why
    This is the paper every transformer question traces back to, and interviewers still ask you to justify its choices, such as the √d_k scaling, multiple heads and the O(n²) cost, in the authors’ own terms.
    What to read
    Read §3.2 closely (3.2.1 scaled dot-product attention, 3.2.2 multi-head attention, 3.2.3 the three places attention is used) with Figure 2. Then §3.5 on sinusoidal positions, §4 with Table 1 on cost per layer, and Table 3 in §6.2 on which variations hurt.
    What to skip
    Skip §5 (training data, hardware, schedule), §6.1 and §6.3 (translation and parsing scores) and the attention visualisations at the end.

Companies that test Transformers most

Put Self-attention from scratch in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan