Skip to content

Why inference is memory-bound: GPU basics for LLM engineers

  • Inference
  • Specialist topic or differentiator
  • About 2 h 30 min to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

A GPU can be slow for three different reasons: it is doing too many FLOPs, it is waiting on memory traffic, or it is waiting on the host to tell it what to do. Arithmetic intensity (FLOPs per byte moved) tells you which one, and the roofline picture turns that into a single threshold per chip. For LLMs the consequence is that prefill is compute-bound while single-token decode is bandwidth-bound: every step reads all the weights plus the KV cache to produce a handful of FLOPs per byte, which is why batching, quantization, and KV-cache compression all help. Interviews ask you to estimate KV-cache memory for a given model and context, to explain why a bigger batch is almost free up to a point, and to say what FlashAttention and tensor parallelism fix at a sentence level.

Builds on

Leads to

You’re ready when you can

What interviewers check, from the resources below.

  • Define the three regimes and give one symptom you would see in each
  • Compute arithmetic intensity for an element-wise op versus a large matmul and say which regime each lands in
  • Explain why fusing two element-wise ops costs about the same as one
  • Describe one way to tell overhead-bound from bandwidth-bound without a profiler
  • Derive KV-cache bytes per token from layers, heads, head dimension and precision, and apply it to a model you know
  • Explain why decoding at batch size 1 is bounded by reading the weights and estimate that time for a 13B model on an A100
  • Compute the bandwidth-to-FLOPs threshold for a given GPU and say what happens above and below it
  • Justify the 2P FLOPs-per-token rule and name what it ignores
  • Explain why prefill is compute-bound and decode is bandwidth-bound in terms of arithmetic intensity
  • State the batch-size threshold for decode to become compute-bound and how int8 weights change it

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Making Deep Learning Go Brrrr From First Principles (opens in a new tab)

    horace.io · Horace He · 40 min · Free · checked 21 Oct 2026

    Why
    The compute / bandwidth / overhead split is the mental model behind every “why is my model slow” question, and this is the clearest short statement of it.
    What to read
    Read “Compute”, then “Bandwidth” including “Reasoning about Memory-Bandwidth Costs” (the factory-and-warehouse picture and operator fusion), then “Overhead”. Stop before the closing remarks.
    What to skip
    The PyTorch-internals asides and the compiler name-dropping; you only need to know that fusion exists and why it helps.

  2. Free alternative

    Transformer Inference Arithmetic (opens in a new tab)

    kipp.ly · kipply (Carol Chen) · 45 min · Free · checked 21 Oct 2026

    Why
    The back-of-envelope formulas interviewers expect you to reproduce on a whiteboard: bytes per token of KV cache, 2P FLOPs per token, and the batch size where an A100 flips from bandwidth-bound to compute-bound.
    What to read
    Read “kv cache”, “capacity”, “latency calculations” and “batch sizes” in that order, then “flops counting”. Do “exercises” at the end if you have time.
    What to skip
    “model parallelism” (awareness only; the scaling-book deep dive covers it better), “intermediate memory costs” and “comparing against real benchmarks”.

  3. Go deeper

    All About Transformer Inference (How To Scale Your Model) (opens in a new tab)

    jax-ml.github.io · Jacob Austin, Sholto Douglas et al. (Google DeepMind) · 70 min · Free · checked 21 Oct 2026

    Why
    The most rigorous free treatment of prefill vs decode rooflines, KV-cache sizing and the latency/throughput trade-off, plus the sharding vocabulary that inference-role interviews expect at awareness level.
    What to read
    Read all of “The Basics of Transformer Inference” (especially “Linear operations: what bottlenecks us?”, “What about attention?”, “What about memory?” and “Modeling throughput and latency for LLaMA 2-13B”), then “Tricks for Improving Generation Throughput and Latency”. Skim “Distributing Inference Over Multiple Accelerators” and “Designing an Effective Inference Engine” for vocabulary only. Try two of the “Worked Problems”. If rooflines are new, read the book's “All About Rooflines” chapter first.
    What to skip
    The appendices, the JetStream walkthrough, and TPU-specific topology details.

Companies that test Inference most

Put Why inference is memory-bound in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan