Skip to content

Transformer math: parameters, FLOPs and memory estimates

  • Inference
  • Asked often
  • About 1 h 30 min to learn
  • 2 vetted resources
Build a plan that includes this

What this topic covers

Applied AI interviews often include a quick sizing question: how many GPUs, how much memory, how many tokens per second. The core rules of thumb are about 2N FLOPs per token for a forward pass, about 6N per token for training, 2 bytes per parameter for bf16 weights and roughly 16 bytes per parameter for mixed-precision Adam training before activations. KV-cache size is 2 × layers × KV heads × head dimension × bytes, per token and per sequence. A strong answer works a real example out loud, such as whether a 70B model fits on two 80 GB H100s and how much room is left for the KV cache. It also knows where the estimate breaks down.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Can state training compute as about 6 × parameters × tokens
  • Can size weights at 2 bytes per parameter in bf16, 1 at int8 and 0.5 at int4
  • Can break mixed-precision Adam training into roughly 16 bytes per parameter before activations
  • Can say whether a 70B model fits on 2×H100 at bf16 and what the KV cache adds
  • Can count a matmul as 2·m·n·k FLOPs and derive about 2N FLOPs per token forward
  • Can explain why training costs about three times the forward pass
  • Can estimate when attention FLOPs start to rival the MLP as context grows
  • Can size the KV cache per token from layers, KV heads and head dimension

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Transformer Math 101 (opens in a new tab)

    EleutherAI Blog · Quentin Anthony, Stella Biderman, Hailey Schoelkopf · 40 min · Free · checked 21 Oct 2026

    Why
    “Will this model fit on two H100s?” and “how much compute does this training run need?” are standard sizing questions. This post gives the few rules of thumb those answers rest on.
    What to read
    Read Compute Requirements and the whole Memory Requirements section: Inference (Model Weights, Total Inference Memory) and Training (Model Parameters, Optimizer States, Gradients, Activations and Batch Size, Total Training Memory). Then do the 70B-on-2×H100 sum yourself: about 140 GB of bf16 weights against 160 GB of HBM, before adding the KV cache.
    What to skip
    Skim Distributed Training (Sharded Optimizers, 3D Parallelism) unless the role is training-heavy. You can also skim the activation-memory formulas.

  2. Free alternative

    All the Transformer Math You Need to Know (opens in a new tab)

    How To Scale Your Model (Google DeepMind) · Jacob Austin, Sholto Douglas, Roy Frostig, Anselm Levskaya, et al. · 60 min · Free · checked 21 Oct 2026

    Why
    Frontier-lab interviewers often expect you to derive the 2N and 6N rules from matmul counts instead of reciting them. This chapter shows how.
    What to read
    Read Counting Dots (including Forward and reverse FLOPs), then Global FLOPs and Params Calculation through General rule of thumb for Transformer FLOPs and Fractional cost of attention with context length. Then read Key-Value (KV) caching under Miscellaneous Math. Try two of A Few Problems to Work before reading the answers.
    What to skip
    On a first pass, skip Gradient checkpointing and the Sparsity and Mixture-of-Experts subsection, which the MoE topic covers. Appendix A on Flash Attention is optional here.

Companies that test Inference most

Put Transformer math in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan