Decoding: greedy, beam, temperature, top-k/top-p
- Inference
- Asked in most interview loops
- About 1 h 35 min to learn
- 3 vetted resources
What this topic covers
The model outputs a probability distribution over the vocabulary, and the decoding strategy decides which token gets picked from it. Greedy and beam search are deterministic and look for high-probability sequences, but on open-ended text they tend to repeat themselves. Sampling adds variety: temperature reshapes the distribution, while top-k and top-p cut it down to a candidate pool before drawing. Interviews ask you to explain or code these knobs, pick settings for a use case (say, extraction vs creative writing), explain why temperature 0 can still give different outputs, and describe stopping conditions and best-of-n sampling.
You’re ready when you can
What interviewers check, from the resources below.
- Explain why greedy decoding can miss a higher-probability sequence and how beam search fixes that
- Explain why beam search does poorly on open-ended generation, and what n-gram penalties do
- Show how temperature changes the softmax, including what happens near 0 and above 1
- Compare top-k with top-p, and say why top-p adapts to how confident the model is
- Explain temperature as rescaling the logits, and what setting it to 0 means
- Pick settings for a factual extraction task and for a brainstorming task, and justify them
- Describe two stopping conditions and how they affect cost and latency
- Explain best-of-n generation and two ways to choose the winning output
- Implement temperature scaling followed by multinomial sampling over a logits vector
- Implement top-k filtering by masking logits to -inf before the softmax
What to study, in order
Each link opened and checked by hand; the date says when.
Start here
How to generate text: using different decoding methods for language generation with Transformers (opens in a new tab)
Hugging Face blog · Patrick von Platen · 30 min · Free · checked 21 Oct 2026
- Why
- “What do temperature and top-p actually do?” is a standard warm-up question, and this post builds each method on the previous one with runnable examples.
- What to read
- Read “Greedy Search” through “Top-p (nucleus) sampling”, then the “Conclusion”.
- What to skip
- The “Appendix” links. Don't try to match the printed GPT-2 outputs word for word, and expect a couple of typos in the code comments.
Free alternative
Generation configurations: temperature, top-k, top-p, and test time compute (opens in a new tab)
Chip Huyen's blog · Chip Huyen · 25 min · Free · checked 21 Oct 2026
- Why
- It ties the sampling knobs to product decisions like latency, cost and consistency, which is how applied interviews frame the question.
- What to read
- Read “Sampling” (with “Temperature”, “Top-k”, “Top-p” and “Stopping condition”), then “Test Time Compute”.
- What to skip
- “Structured Outputs” and its subsections, which belong to a separate topic. Come back to them for constrained decoding.
Go deeper
LLMs from Scratch, Chapter 5 notebook: Decoding strategies to control randomness (opens in a new tab)
rasbt/LLMs-from-scratch · Sebastian Raschka · 40 min · Free · checked 21 Oct 2026
- Why
- Being able to write temperature and top-k sampling over raw logits in a few lines is a common from-scratch coding check.
- What to read
- Work through “5.3 Decoding strategies to control randomness” (5.3.1 to 5.3.3). Run the setup cells in 5.1.1 first so the model and generate function exist.
- What to skip
- “5.1.2” to “5.2” (loss and training), “5.4” and “5.5”. Just run those cells if a later cell depends on them.
Companies that test Inference most
Put Decoding in your plan
Tell us your interview date and hours a day; it lands on the right day, in the right order.