Skip to content

Speculative decoding and other inference accelerations

  • Inference
  • Specialist topic or differentiator
  • About 1 h 30 min to learn
  • 4 vetted resources
Build a plan that includes this

What this topic covers

Decoding is memory-bandwidth bound: each new token re-reads every weight for a single step of work, so the GPU sits mostly idle. Speculative decoding fills that gap by letting a cheap draft (a small model, n-gram lookup, or an extra head like EAGLE/MTP) propose several tokens, then verifying them all in one forward pass of the big model. A rejection-sampling rule keeps the output distribution identical to plain sampling from the target, so the speedup is lossless. Interviews ask why it works, what controls the speedup (draft acceptance rate and draft length), when it stops helping (high batch sizes, compute-bound regimes), and how it relates to KV caching and batching.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Explain why one decoding step is bound by weight loading, not arithmetic
  • Describe how a draft model's candidates are verified in a single forward pass of the main model
  • State why the draft must share the target's tokenizer
  • Explain how the number of requested candidate tokens adapts to the acceptance rate
  • State the two observations that make speculative decoding work
  • Explain the analogy to speculative execution in CPUs and where it breaks for probabilistic outputs
  • Explain why the output distribution is unchanged even though a small model drafted the tokens
  • Write out the accept/reject rule for a drafted token given p(x) and q(x) and the resampling distribution on rejection
  • Derive the expected number of tokens accepted per iteration in terms of α and γ
  • Explain how α (acceptance rate) and the draft-to-target cost ratio determine the wall-time speedup

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Assisted Generation: a new direction toward low-latency text generation (opens in a new tab)

    Hugging Face Blog · Joao Gante · 25 min · Free · checked 21 Oct 2026

    Why
    The standard follow-up to “why is generation slow?” is “so how would you speed it up?” and this post walks the exact chain of reasoning: memory-bound decoding, draft-and-verify, lossless sampling.
    What to read
    Read “Understanding text generation latency” through “Sample with assisted generation”.
    What to skip
    “Future directions” and “Related Work” unless you want the pointers; the code snippets are optional on a first pass.

  2. Free alternative

    Looking back at speculative decoding (opens in a new tab)

    Google Research Blog · Yaniv Leviathan, Matan Kalman, Yossi Matias · 15 min · Free · checked 21 Oct 2026

    Why
    The inventors' own plain-language retrospective: the two observations behind the idea and the CPU speculative-execution analogy that interviewers like to hear.
    What to read
    Read “Background” (both observations) through “Speculative decoding”.
    What to skip
    “Towards more efficient AI” and the acknowledgements.

  3. Go deeper

    Fast Inference from Transformers via Speculative Decoding (opens in a new tab)

    arXiv · Yaniv Leviathan, Matan Kalman, Yossi Matias · 45 min · Free · checked 21 Oct 2026

    Why
    Inference-focused rounds ask you to derive the expected speedup and the acceptance rule, and this is the paper those questions are lifted from.
    What to read
    Read “2 Speculative Decoding” (2.1 to 2.3) and “3 Analysis” sections 3.1 “Number of Generated Tokens” through 3.3 “Walltime Improvement” and 3.5 “Choosing γ”.
    What to skip
    3.4 arithmetic-operation counts, “4 Experiments”, and the appendix proofs unless you are asked to prove losslessness.

  4. Go deeper

    gpt-fast (opens in a new tab)

    GitHub · meta-pytorch · 45 min · Free · checked 22 Oct 2026

    Why
    gpt-fast shows speculative decoding, int8/int4 weight-only quantization and tensor parallelism in under 1,000 lines of plain PyTorch, so you can explain the accept/reject step from real code rather than a diagram. It changes rarely now (last push August 2025) but remains the standard minimal reference.
    What to read
    Read generate.py: prefill, decode_one_token, decode_n_tokens and especially speculative_decode (draft k tokens, verify in one target pass, accept or resample). Then read the README’s “Speculative Sampling” and “Quantization” sections and skim quantize.py.
    What to skip
    Skip tp.py, GPTQ.py, the AMD and Mixtral notes and the model download steps unless you have the GPUs to run it.

Companies that test Inference most

Put Speculative decoding and other inference accelerations in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan