Skip to content

Multi-head attention and positional encoding

  • Transformers
  • Asked often
  • About 2 h to learn
  • 4 vetted resources
Build a plan that includes this

What this topic covers

Multi-head attention splits the model dimension into several smaller attention heads that run in parallel, so each head can attend to a different kind of relationship, and their outputs are concatenated and projected back. Attention alone is order-blind, so position has to be injected: absolute sinusoidal or learned embeddings were the original answer, and rotary position embeddings (RoPE) are what nearly every modern decoder uses because they turn position into a rotation of queries and keys that makes attention scores depend on relative distance. Interviews ask you to implement multi-head attention from scratch with correct tensor shapes, explain why heads help and what they cost, and say what RoPE does differently from sinusoidal encodings and why it matters for long-context extension.

Builds on

Leads to

You’re ready when you can

What interviewers check, from the resources below.

  • Explain why a self-attention layer with no positional signal is permutation-invariant
  • List the properties a good positional encoding should have and say which ones sinusoidal encodings fail
  • Describe RoPE as a position-dependent rotation of query/key pairs and why that makes scores depend on relative offset
  • State why RoPE leaves the token's semantic content (vector norm) unchanged
  • Implement the split into heads, per-head scaled dot-product attention, and the concatenate-and-project step with correct shapes
  • Explain why the scaling factor uses the per-head dimension, not the model dimension
  • State the parameter count and FLOP cost of multi-head versus single-head attention at the same model width
  • Describe how a causal mask would be added to your implementation
  • Derive why the dot product of two rotated vectors depends only on the difference of their positions
  • Explain how the per-pair rotation frequencies are chosen and what low versus high frequencies capture

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    You could have designed state of the art positional encoding (opens in a new tab)

    Hugging Face Blog · Christopher Fleetwood · 35 min · Free · checked 21 Oct 2026

    Why
    Interviewers probe whether you understand why attention needs position information and what RoPE actually does to queries and keys, not just that “Llama uses RoPE”.
    What to read
    Read “Problem Statement” through “Rotary Postional Encoding”; the “Desirable Properties” list is the mental checklist to carry into the interview.
    What to skip
    “Extending RoPE to n-Dimensions” and “The future of positional encoding” unless you work on vision or multimodal models.

  2. Free alternative

    Implement Multi-Head Attention (opens in a new tab)

    Deep-ML · Deep-ML · 40 min · Free · checked 21 Oct 2026

    Why
    “Implement multi-head attention in NumPy/PyTorch” is one of the most common live-coding asks for applied LLM roles, and the shape handling is where candidates stumble.
    What to read
    Solve the problem end to end in the browser editor; then re-do it from a blank file without looking at your first attempt.
    What to skip
    Nothing; it is one problem. Do not reach for a framework's built-in attention layer.

  3. Go deeper

    Rotary Embeddings: A Relative Revolution (opens in a new tab)

    EleutherAI Blog · Stella Biderman, Sid Black, Charles Foster, Leo Gao, Eric Hallahan, Horace He, Ben Wang, Phil Wang · 45 min · Free · checked 21 Oct 2026

    Why
    Senior-level follow-ups ask you to derive why a rotation gives relative position and to sketch the implementation; this is the post that popularised RoPE in open models.
    What to read
    Read “What’s the Problem?” through “Derivation”, then “How is this different from the sinusoidal embeddings used in “Attention is All You Need”?” and “Implementation”.
    What to skip
    “Experiments” onward (benchmark tables for Performer, billion-parameter runs and runtime numbers) unless asked about evidence.

  4. Go deeper

    RoFormer: Enhanced Transformer with Rotary Position Embedding (opens in a new tab)

    arXiv · Jianlin Su et al. · 35 min · Free · checked 22 Oct 2026

    Why
    RoPE is the positional scheme in almost every modern open LLM, and interviewers ask you to explain why rotating queries and keys encodes relative position.
    What to read
    Read §2.2–2.3 for the absolute and relative baselines, then §3.2 on the rotary formulation (the 2D case first, then the general block-diagonal form) with Figure 1, and §3.3 on its properties, including long-term decay and use with linear attention.
    What to skip
    Skip §3.4’s theoretical derivation, §4.1, §4.3 and §4.5 experiments, and treat §4.2 as a skim.

Companies that test Transformers most

Put Multi-head attention and positional encoding in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan