Skip to content

Quantization for LLM inference

  • Inference
  • Asked often
  • About 1 h 30 min to learn
  • 4 vetted resources
Build a plan that includes this

What this topic covers

Quantization stores a model's weights (and sometimes activations) in fewer bits, such as int8 or 4-bit, so the model fits in less GPU memory and moves less data per token. The core idea is mapping a float range onto a small integer grid with a scale (and, for asymmetric schemes, a zero point), then accepting some rounding error. Interviews ask you to estimate memory for a model at fp16 vs int8 vs 4-bit, explain why outlier values make naive quantization fail, and contrast post-training methods (GPTQ, AWQ, GGUF, bitsandbytes) with quantization-aware training. Expect a deployment question too: which method would you pick for a given GPU, latency target, or need to fine-tune with LoRA.

Builds on

Leads to

You’re ready when you can

What interviewers check, from the resources below.

  • Calculate the weight memory of a 7B model at fp32, fp16, int8 and 4-bit
  • Explain symmetric (absmax) vs asymmetric (zero-point) quantization and when each fits better
  • Describe why clipping the range and calibrating on sample data reduce error
  • Contrast dynamic and static quantization of activations
  • Explain at a high level what GPTQ and GGUF each do and where you'd use them
  • Compare fp32, fp16 and bf16 by range and precision
  • Work through absmax and zero-point quantization of a small vector by hand
  • Explain what outlier features are and why they appear in larger models
  • Describe how LLM.int8() splits a matmul into an int8 part and an fp16 part
  • Name which methods quantize on the fly and which need a calibration pass

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    A Visual Guide to Quantization (opens in a new tab)

    Exploring Language Models (newsletter) · Maarten Grootendorst · 50 min · Free · checked 21 Oct 2026

    Why
    Serving cost questions almost always reach “how would you fit this model on a smaller GPU?”, and quantization is the first answer interviewers expect you to explain properly.
    What to read
    Read “Part 1: The ‘Problem’ with LLMs” through “Part 3: Post-Training Quantization”, ending after “The Realm of 4-bit Quantization” (GPTQ and GGUF).
    What to skip
    “Part 4: Quantization Aware Training” (BitNet and 1.58-bit models) unless you're interviewing for an inference or model-efficiency team.

  2. Free alternative

    A Gentle Introduction to 8-bit Matrix Multiplication for transformers at scale using Hugging Face Transformers, Accelerate and bitsandbytes (opens in a new tab)

    Hugging Face Blog · Younes Belkada, Tim Dettmers · 40 min · Free · checked 21 Oct 2026

    Why
    Covers the same int8 basics from the people who built LLM.int8(), plus the outlier problem that explains why naive quantization breaks large models.
    What to read
    Read “Common data types used in Machine Learning” and “Introduction to model quantization”, then the LLM.int8() summary through “What does 0 degradation mean?”.
    What to skip
    From “Hugging Face transformers integration nuances” onward (2022-era setup, hardware notes and roadmap are dated).

  3. Go deeper

    Quantization overview (Transformers docs) (opens in a new tab)

    Hugging Face Transformers docs · Hugging Face · 20 min · Free · checked 21 Oct 2026

    Why
    Turns theory into the deployment choice interviewers ask about: which method works on your hardware, quantizes on the fly, compiles, or still lets you fine-tune with PEFT.
    What to read
    Read the “Overview” intro, try the quantization picker with one model and GPU, then study the full compatibility table, focusing on the bitsandbytes, AWQ, GPT-QModel, GGUF and torchao rows.
    What to skip
    “Resources” (external courses) and “User-Friendly Quantization Tools”, plus the individual method pages unless a role names a specific stack.

  4. Go deeper

    llama.cpp (opens in a new tab)

    GitHub · ggml-org · 60 min · Free · checked 22 Oct 2026

    Why
    llama.cpp and its GGUF format are how most people run quantized models locally, and “what does Q4_K_M actually mean?” is a common follow-up once you mention quantization.
    What to read
    Read tools/quantize/README.md (“Overview”, “Quantize the GGUF”, the examples using --imatrix and per-tensor types, “Memory/Disk Requirements”). Then read tools/perplexity/README.md’s quantization comparison tables. Build with docs/build.md, quantize a small GGUF to Q4_K_M with llama-quantize, and compare speed with tools/llama-bench.
    What to skip
    Skip the backend list in the main README, ggml/src kernels, and the multimodal conversion steps in the quantize README.

Companies that test Inference most

Put Quantization for LLM inference in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan