Skip to content

KV cache

  • Inference
  • Asked often
  • About 1 h 30 min to learn
  • 4 vetted resources
Build a plan that includes this

What this topic covers

When a model generates text one token at a time, the keys and values of earlier tokens never change. The KV cache stores them so each new step only computes the newest token, turning repeated quadratic work into linear work. The price is memory that grows with sequence length, which is why serving systems page, share or evict the cache.

Builds on

Leads to

You’re ready when you can

What interviewers check, from the resources below.

  • Explain why past keys and values never change during generation
  • State the compute saving (quadratic → linear per step) and the memory cost
  • Know when the cache must be cleared
  • Point to where the cache buffers are registered and appended
  • Explain the use_cache flag and the reset method
  • Explain why the speedup can vanish for a tiny model on a GPU
  • Explain why naive KV cache allocation wastes memory
  • Describe PagedAttention with the virtual-memory analogy
  • Name one benefit of sharing cache blocks between sequences
  • Explain how a radix tree over token ids lets two requests share KV cache

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Understanding and Coding the KV Cache in LLMs from Scratch (opens in a new tab)

    Ahead of AI · Sebastian Raschka · 25 min · Free · checked 20 Oct 2026

    Why
    KV caching is the first thing asked after “implement attention”: why generation is slow and how to fix it.
    What to read
    Read “What Is a KV Cache?” through “KV cache Advantages and Disadvantages”.
    What to skip
    The Qwen3 / Llama 3 bonus unless you're targeting an inference role.

  2. Free alternative

    LLMs-from-scratch: KV cache bonus code (opens in a new tab)

    GitHub · rasbt/LLMs-from-scratch · Sebastian Raschka · 45 min · Free · checked 20 Oct 2026

    Why
    Runnable before/after code: the fastest way to make the KV cache concrete.
    What to read
    Diff gpt_ch04.py against gpt_with_kv_cache.py; read the README's speed and trade-off notes.
    What to skip
    gpt_with_kv_cache_optimized.py on a first pass.

  3. Go deeper

    vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (opens in a new tab)

    vLLM blog · Woosuk Kwon, Zhuohan Li et al. · 15 min · Free · checked 20 Oct 2026

    Why
    Shows what happens to the KV cache at serving scale: memory fragmentation, and paging as the fix.
    What to read
    Read “The Secret Sauce: PagedAttention”.
    What to skip
    The benchmarks and the Vicuna / Chatbot Arena story.

  4. Go deeper

    SGLang (opens in a new tab)

    GitHub · sgl-project · 45 min · Free · checked 22 Oct 2026

    Why
    SGLang is the other major open serving engine, and its radix-tree KV cache is the standard answer to “how do you reuse KV cache across requests that share a prefix?”
    What to read
    Read python/sglang/srt/mem_cache/radix_cache.py: TreeNode, RadixCache.match_prefix, insert, _split_node, evict and the inc_lock_ref / dec_lock_ref pair. Then skim python/sglang/srt/managers/schedule_policy.py to see how cache hits shape scheduling order. If you have a GPU, follow the README’s Get Started and the linked Quickstart to launch a model.
    What to skip
    Skip the rest of srt (model runners, kernels, disaggregation, multimodal) and the ecosystem section of the README.

Companies that test Inference most

Put KV cache in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan