Skip to content

Batching, continuous batching and LLM serving

  • Inference
  • Asked often
  • About 80 min to learn
  • 4 vetted resources
Build a plan that includes this

What this topic covers

LLM serving is dominated by the decode phase, where each step reads all model weights to produce one token per sequence, so a single request leaves the GPU memory-bound and mostly idle. Batching many sequences per step fixes that, but static batching waits for the longest sequence to finish; continuous (iteration-level) batching swaps finished sequences out and new ones in at every step, and PagedAttention makes the KV cache compact enough to batch widely. Interviews ask why static batching wastes GPU time, what TTFT versus tokens-per-second each depend on (prefill versus decode), how to size the KV cache to find the maximum batch, and when adding batch size stops improving throughput and only adds latency.

Builds on

Leads to

You’re ready when you can

What interviewers check, from the resources below.

  • Explain why static batching wastes GPU time when sequences in a batch finish at different lengths
  • Describe iteration-level scheduling and when a new request can join a running batch
  • Explain how PagedAttention's block-based KV cache lets more sequences fit in one batch
  • State the effect of continuous batching on throughput and on p50 latency, and why both improve
  • Define time-to-first-token and tokens-per-second and say which inference phase drives each
  • Explain why prefill is one forward pass over the prompt and decode is one pass per generated token
  • Read a throughput-versus-latency chart and point to where adding batch size stops being free
  • Compute a GPU's ops:byte ratio from its FLOPS and memory bandwidth
  • Explain why single-sequence decode is memory-bound and what batching changes about that
  • Size the KV cache per token for a given model and work out how many sequences fit in the GPU memory left after weights

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    How continuous batching enables 23x throughput in LLM inference while reducing p50 latency (opens in a new tab)

    Anyscale Blog · Cade Daniel, Chen Shen, Eric Liang, Richard Liaw · 25 min · Free · checked 21 Oct 2026

    Why
    Continuous batching is the one idea behind why vLLM and TGI beat naive serving, and “why is static batching bad for LLMs?” is a near-guaranteed inference question.
    What to read
    Read “The basics of LLM inference” through “PagedAttention and vLLM”, then jump to “Benchmarking results: Throughput” and “Benchmarking results: Latency” for the charts only.
    What to skip
    “Benchmarking setup”, “Experiments”, “Hardware/model”, “Frameworks” and the Ray sections at the end; the numbers are mid-2023 and every framework in them has changed since. Keep the concepts, not the 23x.

  2. Free alternative

    Benchmarking Text Generation Inference (opens in a new tab)

    Hugging Face Blog · Derek Thomas · 20 min · Free · checked 21 Oct 2026

    Why
    Gives you the serving vocabulary (TTFT, prefill versus decode, the batch-size knee) with real throughput-versus-latency charts, so you can reason about trade-offs out loud.
    What to read
    Read “Latency vs Throughput” and “Pre-filling and Decoding” under “Pre-requisites”, then “Understanding the Benchmarking tool”.
    What to skip
    “Setup”, “Getting Started”, “Main Components” and “Winding Down” unless you plan to run the tool yourself.

  3. Go deeper

    A guide to LLM inference and performance (opens in a new tab)

    Baseten Blog · Varun Shenoy, Philip Kiely · 35 min · Free · checked 21 Oct 2026

    Why
    Senior inference questions go past “use vLLM” to “is this workload compute- or memory-bound, and how many sequences fit?”; this is the napkin math for that.
    What to read
    Read “Calculating the operations per byte (ops:byte) ratio” through “Batching memory-bound processes on a GPU”, then “Prefilling with batched prompt tokens on each GPU” and “Estimating total generation time on each GPU”.
    What to skip
    “Reading GPU specs” and “A10 key specs” if you can read a spec sheet, and “Benchmarks for Llama 2 7B Chat on NVIDIA GPUs” beyond a glance.

  4. Go deeper

    Nano-vLLM (opens in a new tab)

    GitHub · GeeeekExplorer · 60 min · Free · checked 22 Oct 2026

    Why
    A vLLM-style engine in about 1,200 lines of Python, with paged KV blocks, prefix caching and continuous batching you can read in one sitting. It turns “how does continuous batching work?” into code you can point at.
    What to read
    Read nanovllm/engine/scheduler.py (schedule, preempt, postprocess), then nanovllm/engine/block_manager.py (block allocation, hash-based prefix reuse, can_allocate) and nanovllm/engine/llm_engine.py for the step loop. Run example.py with Qwen3-0.6B if you have a CUDA GPU; bench.py compares throughput.
    What to skip
    Skip nanovllm/layers and models/qwen3.py beyond a glance, and the tensor-parallel and CUDA-graph code in model_runner.py.

Companies that test Inference most

Put Batching, continuous batching and LLM serving in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan