Skip to content

Design an LLM inference and batching API

  • System design
  • Asked often
  • About 2 h 30 min to learn
  • 4 vetted resources
Build a plan that includes this

What this topic covers

You design the service that sits in front of the GPUs: a request queue, a scheduler that batches work, and an API that streams tokens back. Interviewers ask it because it shows whether you understand where LLM latency and cost actually come from. A strong answer separates prefill from decode, explains why continuous batching beats static batching, and sets targets for time to first token and time per output token. It also covers admission control and per-tenant quotas so one customer can’t starve the rest, and it does the rough GPU maths: weights plus KV cache per request against memory, and memory bandwidth against tokens per second.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Defines time to first token and time per output token, and says which one prefill and decode drive
  • Explains why continuous batching gives more throughput than static batching
  • Estimates decode speed from model size and memory bandwidth
  • Works out KV cache memory per request and how it caps batch size
  • Adds admission control and per-tenant quotas to the design
  • Can tell time per output token apart from inter-token latency
  • Explains why P99 matters more than the mean for a public API
  • Defines goodput and writes an example service level for time to first token
  • Picks the main metric for a chat product versus a batch job
  • Explains how fragmentation in the KV cache limits batch size

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    LLM Inference Performance Engineering: Best Practices (opens in a new tab)

    Databricks · Megha Agarwal, Asfandyar Qureshi, Nikhil Sardana, Linden Li, Julian Quevedo, Daya Khudia · 45 min · Free · checked 21 Oct 2026

    Why
    An inference service design lives or dies on the numbers. This post gives you the metrics, the batching trade-off and the memory-bandwidth maths that an interviewer expects you to reach for.
    What to read
    Read the sections on which metrics matter, why memory bandwidth is the bottleneck, Model Bandwidth Utilization (work through the 7B example yourself), the batching comparison in the benchmarks, and the closing recommendations. Then sketch your own API: queue, scheduler, streaming, and per-tenant limits.
    What to skip
    Skim the MPT-7B and Llama2-70B benchmark charts once you have the shape of the results, and the FAQ.

  2. Free alternative

    Key metrics for LLM inference (opens in a new tab)

    LLM Inference Handbook (Modular) · 20 min · Free · checked 21 Oct 2026

    Why
    Latency targets are the first thing to agree in this design. This short page gives you the vocabulary to set them, including goodput, which ties throughput to your service levels.
    What to read
    Read Latency, the section on mean, median and P99, Throughput, Goodput, and the latency versus throughput trade-offs. Try the latency playground on the page.
    What to skip
    Skip the additional resources list at the end.

  3. Go deeper

    Efficient Memory Management for Large Language Model Serving with PagedAttention (opens in a new tab)

    arXiv · Woosuk Kwon et al. · 60 min · Free · checked 21 Oct 2026

    Why
    If the interviewer pushes on how the scheduler fits more requests on one GPU, this is the paper behind the answer most serving stacks use today.
    What to read
    Read section 1 (introduction), 2.3 (batching techniques for LLMs), 3 (memory challenges in LLM serving), 4.1 and 4.2 (PagedAttention and the KV cache manager) and 4.5 (scheduling and preemption). Glance at 7.3 on recomputation versus swapping.
    What to skip
    Skip section 5 (implementation and kernels), 4.6 (distributed execution) and most of the evaluation in section 6; the headline throughput gain is enough.

  4. Go deeper

    vLLM (opens in a new tab)

    GitHub · vllm-project · 75 min · Free · checked 22 Oct 2026

    Why
    vLLM is the default open-source serving engine, and inference-design rounds often probe whether you know how it is put together: API server, engine core, workers, scheduler and KV cache manager.
    What to read
    Read docs/design/arch_overview.md (“Entrypoints” and “V1 Process Architecture”) and docs/design/prefix_caching.md (“Data Structure”, “Block Allocation”, “Eviction (LRU)”). Then open vllm/v1/core/sched/scheduler.py at Scheduler.schedule() and vllm/v1/core/kv_cache_manager.py. If you have a GPU, install with uv pip install vllm and follow the docs Quickstart to serve a small model behind the OpenAI-compatible API.
    What to skip
    Skip csrc kernels and docs/design/paged_attention.md, which the repo itself marks as historical. Skip model-specific code, quantization backends and distributed parallelism.

Companies that test System design most

Put Design an LLM inference and batching API in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan