Skip to content

The decoder-only transformer (GPT forward pass)

  • Transformers
  • Asked in most interview loops
  • About 2 h to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

A GPT-style model is a stack of identical blocks sitting between a token embedding layer and a final projection back to vocabulary logits. Each block applies causal (masked) multi-head attention and a feed-forward MLP, each wrapped in layer normalization and a residual connection. Interviews expect you to trace tensor shapes from token IDs to next-token probabilities, explain why the causal mask makes training parallel but generation sequential, and count parameters roughly. Senior loops follow up with what modern models changed since GPT-2: RMSNorm and pre-norm placement, RoPE, SwiGLU, grouped-query or latent attention, and mixture-of-experts layers.

Builds on

Leads to

You’re ready when you can

What interviewers check, from the resources below.

  • Trace shapes from token IDs (batch, seq) to logits (batch, seq, vocab)
  • Implement a transformer block with pre-norm, causal multi-head attention, GELU MLP and two residual connections
  • Explain why the shortcut connections and layer norm matter for training deep stacks
  • Estimate GPT-2 small's parameter count and say where most parameters live
  • Write a greedy generation loop that crops the context to the model's maximum length
  • Contrast encoder, decoder and decoder-only blocks
  • Draw the causal attention mask and explain what it blocks
  • List the weight matrices in one GPT-2 block and their shapes
  • Explain how the final hidden state becomes a next-token distribution
  • Explain what multi-head latent attention and grouped-query attention save compared with standard multi-head attention

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Chapter 4: Implementing a GPT model from Scratch To Generate Text (opens in a new tab)

    LLMs-from-scratch (GitHub) · Sebastian Raschka · 90 min · Free · checked 21 Oct 2026

    Why
    “Write a GPT block” and “walk me through the forward pass with shapes” are standard asks once attention is covered, and this notebook builds exactly that piece by piece.
    What to read
    Work through ch04.ipynb from “4.1 Coding an LLM architecture” to “4.7 Generating text”, running each cell and printing tensor shapes as you go.
    What to skip
    gpt.py and previous_chapters.py are just packaged copies of the same code; the extra GPT-2 medium/large/XL configs in 4.6 unless you want the parameter-count practice.

  2. Free alternative

    The Illustrated GPT-2 (Visualizing Transformer Language Models) (opens in a new tab)

    jalammar.github.io · Jay Alammar · 50 min · Free · checked 21 Oct 2026

    Why
    A picture-first walk through the GPT-2 forward pass for anyone who wants the mental model before (or instead of) writing the code.
    What to read
    Read Part 1 from “The Evolution of the Transformer Block” to “End of part #1”, then Part 2 from “The Illustrated Masked Self-Attention” to “You've Made It!”.
    What to skip
    Part 2's opening “Self-Attention (without masking)” if you already did the self-attention topic, and Part 3 “Beyond Language Modeling”.

  3. Go deeper

    The Big LLM Architecture Comparison (opens in a new tab)

    Ahead of AI · Sebastian Raschka · 60 min · Free · checked 21 Oct 2026

    Why
    Senior interviewers follow “explain GPT-2” with “what's different in a 2026 open model?”, and this maps each modern tweak onto the GPT-2 baseline.
    What to read
    Read sections 1–3 (“DeepSeek V3/R1” through “3.3 Gemma 3 Summary”), “7.1 No Positional Embeddings (NoPE)”, and section 9 “GPT-OSS”.
    What to skip
    Every other model section (they repeat the same ideas with new numbers) and the linear-attention material in sections 12 and 14 unless you're targeting an inference or research role.

Companies that test Transformers most

Put The decoder-only transformer (GPT forward pass) in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan