LLM and generative AI interview questions
What LLM and generative AI interviews ask about how the models work: attention and the transformer, tokenization, fine-tuning and alignment, and how a model is served, from the KV cache and batching to quantization and speculative decoding.
43 questions across 19 topics. Answer each out loud, then open the points a strong answer covers.
Self-attention from scratch
Asked in most interview loops
Why do we divide attention scores by the square root of the key dimension?
- With unit-variance entries, a query-key dot product has variance d_k, so raw scores spread wider as the dimension grows.
- Large scores push softmax towards a near one-hot output, where gradients become tiny and training stalls.
- Dividing by √d_k brings score variance back to about one, keeping softmax in a range with useful gradients.
How would you implement multi-head causal self-attention, and what does it cost in memory?
- Project the input into Q, K and V, split into h heads, compute softmax(QKᵀ/√d_k)V per head, then concatenate and project back.
- For the causal mask, set scores above the diagonal to minus infinity before softmax, and subtract the row max for numerical stability.
- Each head builds a seq_len by seq_len score matrix, so memory and compute grow as O(n²) in sequence length.
What do queries, keys and values each represent, and why learn separate projections for them?
- The query encodes what a token is looking for, the key encodes what it offers for matching, and the value carries the content passed on when it is attended to.
- Scores come from QKᵀ, so separate query and key projections make attention asymmetric: token i can attend strongly to token j without j attending back to i.
- A separate value projection decouples what is used for matching from what is passed on, so the model can match on one feature and copy a different one.
Your attention code starts returning NaNs once you add a padding mask; what would you check?
- A row where every score is masked to minus infinity makes softmax compute 0 divided by 0, so look for queries, such as padding positions, with nothing left to attend to.
- Mask with the dtype’s minimum finite value rather than a hard-coded −1e9, which overflows in fp16, and zero out or ignore any fully masked rows.
- Also confirm the mask is applied before softmax rather than after, broadcasts to (batch, heads, query, key) as intended, and that padded positions are excluded from the loss.
How does self-attention compare with recurrence for modelling long-range dependencies, and what does it cost?
- Self-attention connects any two positions in one step, so the path between distant tokens has length O(1), while an RNN passes information through O(n) sequential steps.
- Every position is computed at once during training, so attention parallelises across the sequence on GPUs, whereas an RNN must wait for each previous hidden state.
- The price is O(n²·d) compute per layer against O(n·d²) for recurrence, so attention is cheaper only while sequence length stays below the model dimension.
The decoder-only transformer (GPT forward pass)
Asked in most interview loops
Can you walk me through a GPT forward pass from token IDs to next-token probabilities, with shapes?
- Token IDs of shape (batch, seq) index an embedding table, plus position information, giving hidden states of shape (batch, seq, d_model).
- Each block applies pre-norm causal multi-head attention, then an MLP, each with a residual connection; the shape stays (batch, seq, d_model).
- A final norm and projection to the vocabulary give logits (batch, seq, vocab); softmax at the last position gives next-token probabilities.
Why can a decoder-only model train on all positions at once but generate one token at a time?
- The causal mask lets each position see only earlier tokens, so one forward pass computes the next-token loss at every position.
- At inference each new token depends on the one just sampled, so generation needs a separate forward pass per token.
- A KV cache stores past keys and values, so each step computes attention only for the new token instead of the whole prefix.
Roughly how many parameters does a GPT-style model have, and where do they sit?
- Each block’s attention has four d_model by d_model matrices for Q, K, V and the output, about 4d², and an MLP with 4× expansion adds about 8d².
- So L blocks hold about 12·L·d² parameters, while the token embedding adds vocab × d, often shared with the output projection; norms and biases are negligible.
- For GPT-2 small, with 12 layers, d_model 768 and a vocabulary of about 50,000, that gives roughly 85M in blocks plus 39M in embeddings, about 124M in total.
What do modern open decoder-only models change from the GPT-2 design, and why?
- LayerNorm is usually replaced by RMSNorm, which drops mean-centring and the bias for a cheaper normalisation, still placed before each sub-layer as GPT-2 already did.
- Learned absolute position embeddings give way to RoPE applied to queries and keys, and the GELU MLP to a gated SwiGLU MLP, which tends to train better at equal cost.
- Grouped-query or multi-head latent attention shrinks the KV cache for cheaper inference, and some models swap dense MLPs for mixture-of-experts layers to add capacity without more compute per token.
What to study for The decoder-only transformer (GPT forward pass)
Decoding: greedy, beam, temperature, top-k/top-p
Asked in most interview loops
What do temperature and top-p actually do to the next-token distribution?
- Temperature divides the logits before softmax: below 1 sharpens towards the top token, above 1 flattens it, and near 0 approaches greedy.
- Top-k keeps a fixed number of the likeliest tokens; top-p keeps the smallest set whose probabilities sum to p, then renormalises.
- Top-p adapts to confidence: when the model is sure the pool is tiny, and when it’s unsure the pool widens.
Which decoding settings would you pick for extraction versus creative writing, and why?
- For extraction, use greedy or a temperature near 0 so the output stays consistent and close to the most likely answer.
- For creative text, sample with moderate temperature and top-p, since greedy and beam search tend to repeat themselves on open-ended text.
- Even at temperature 0 outputs can vary, because GPU floating-point sums depend on batching and kernels, so near-tied tokens can flip.
How would you implement top-p sampling over raw logits, and which edge cases would you handle?
- Divide the logits by temperature, take a stable softmax, sort probabilities in descending order and take the cumulative sum to find the smallest set reaching p.
- Keep the token that crosses the threshold so at least one always survives, mask the rest, renormalise and sample, then map the sorted index back to its token ID.
- Treat temperature 0 as greedy argmax to avoid dividing by zero, and fix the order of temperature, top-k and top-p explicitly, since the order changes which tokens survive.
KV cache
Asked often
How do you estimate the KV cache memory for a request, and what does grouped-query attention change?
- Per token the cache holds one key and one value for every layer and KV head, so bytes per token = 2 × layers × KV heads × head_dim × bytes per element.
- Multiply by sequence length and batch size: a 32-layer model with 32 KV heads of dimension 128 in fp16 needs 0.5 MiB per token, or 2 GiB at 4,096 tokens.
- Grouped-query attention lets several query heads share one KV head, shrinking the cache by the ratio of query heads to KV heads, for example four times with 32 and 8.
Short requests run fine but long-context ones hit out-of-memory errors; what’s happening and what can you do?
- Weights are a fixed cost, but the KV cache grows linearly with tokens per sequence times concurrent sequences, so a few long contexts can exhaust whatever memory remains.
- Cap the maximum context and batched tokens, use paged allocation so memory isn’t reserved up front for the full maximum length, and let the scheduler preempt sequences instead of crashing.
- To shrink the cache itself, quantize it to fp8 or int8, choose a model with grouped-query attention, or offload cold blocks to CPU memory at some latency cost.
How can a serving system reuse KV cache across different requests, and when is that reuse valid?
- With causal attention a token’s keys and values depend only on the tokens before it, so requests sharing an identical prefix, like a system prompt, can share its cached blocks.
- vLLM hashes full blocks of token IDs together with their prefix to find matches, while SGLang keeps a radix tree over tokens and splits a node where a new request diverges.
- Reuse needs an exact token match from the very start, with the same model and positions; one changed early token invalidates everything after it, so put stable content first.
Tokenization and BPE
Asked often
Walk me through how BPE trains a tokenizer and then encodes new text.
- Training starts from the 256 byte values, counts adjacent token pairs in the corpus, merges the most frequent pair into a new token, and repeats until the vocabulary reaches its target size.
- The tokenizer stores the vocabulary and the ordered merge list; encoding splits text into bytes and repeatedly applies the earliest-learned merge present, and decoding concatenates the bytes back.
- Byte-level BPE can encode any string with no unknown token, and vocabulary size is the main trade-off: more tokens shorten sequences but enlarge the embedding and output layers.
Why does a model struggle to count letters or add long numbers, and why does it cost more in some languages?
- The model sees token IDs, not characters, so a word like “strawberry” may be one or a few tokens and its spelling must be memorised rather than read.
- Numbers get split into irregular chunks by learned merges, so digits do not line up by place value; some tokenizers split numbers into single digits or fixed groups to help.
- Merges are learned mostly from English-heavy data, so scripts like Devanagari, which take several UTF-8 bytes per character, need more tokens per word, raising cost and using up context.
Fine-tuning and LoRA
Asked often
How does LoRA work, and why can it add no inference latency?
- LoRA freezes a weight W of shape d×k and learns ΔW = BA, with B of shape d×r and A of shape r×k, where rank r is far smaller than d and k.
- A is initialised randomly and B at zero, so training starts exactly from the base model, and the update is scaled by α/r so changing the rank needs less retuning.
- Trainable parameters drop from d·k to r(d + k) per matrix, and after training W + (α/r)BA merges into one matrix of the original shape, so serving costs nothing extra.
Your LoRA fine-tune shows falling training loss, but the model seems worse in use; what would you check?
- Look for overfitting, such as validation loss rising while training loss falls or outputs parroting training phrasing, and respond with fewer epochs, a lower learning rate or more varied data.
- Audit the data for errors and inconsistent formatting, confirm inference uses the same chat template as training, and check the loss is masked to response tokens only.
- Compare against the base model on held-out task examples and general-skill evals, since narrow tuning can erode abilities, and mixing in general data or tuning less aggressively can help.
Quantization for LLM inference
Asked often
Why does naive int8 quantization hurt larger models, and how do methods like LLM.int8() and AWQ handle it?
- Absmax scaling maps the largest magnitude to the edge of the int8 range, so a few outlier values stretch the scale and squeeze ordinary values onto very few levels.
- Larger models develop outlier features concentrated in a few hidden dimensions; LLM.int8() keeps those columns in an fp16 matmul and runs the rest of the multiplication in int8.
- AWQ uses activation magnitudes to find the weight channels that matter most and scales them up before rounding, and finer per-channel or per-group scales also limit the damage.
Weight-only 4-bit quantization sped up decoding at batch size 1; why might the gain shrink at large batch sizes?
- At small batch, decode is memory-bandwidth-bound, so reading weights at about a quarter of their fp16 size directly cuts the time spent on each step.
- Weight-only kernels dequantize weights back to 16-bit before the matmul, so the arithmetic is no cheaper, and as batch grows the step turns compute-bound and that overhead shows.
- At large batch the KV cache can also outweigh the weights, so for throughput consider fp8 or int8 weight-and-activation schemes on low-precision tensor cores, plus KV cache quantization.
Multi-head attention and positional encoding
Asked often
Why does attention need positional information, and how does RoPE differ from sinusoidal encodings?
- Without a positional signal, unmasked self-attention is permutation-equivariant: shuffling the input tokens only shuffles the outputs, so the model cannot tell word order.
- Sinusoidal encodings add a fixed vector of sines and cosines at different frequencies to the token embeddings once, at the input, mixing position into the content vector.
- RoPE rotates each pair of query and key dimensions by an angle proportional to position at every layer, so the q·k score depends only on content and relative offset, leaving values untouched.
Why split attention into several heads instead of one head of the same width, and what does that cost?
- Each head has its own projections into a d_model/h subspace and its own softmax, so different heads can track different relationships, such as the previous token or a related noun.
- At the same model width, parameters and projection FLOPs match single-head attention, but there are h score matrices of n by n, so score memory grows with the head count.
- Each head’s scores are divided by √(d_model/h), too many heads leave each one too narrow to be useful, and the KV cache grows with the number of K/V heads, which motivates GQA.
What to study for Multi-head attention and positional encoding
Why does static batching waste GPU time in LLM serving, and how does continuous batching fix it?
- In a static batch every sequence waits for the longest one, so slots for finished sequences sit idle and new requests queue until the whole batch completes.
- Continuous batching schedules at the level of each decode step, removing finished sequences and admitting waiting requests at every iteration, so the batch stays full.
- It needs a KV cache that grows and shrinks per sequence without fragmentation, which PagedAttention’s fixed-size blocks provide, letting more sequences fit in memory at once.
After you raised the maximum batch size, throughput rose but p99 latency spiked; what do you check?
- Decode is nearly free to batch while memory-bound, but past the knee each step takes longer, so plot inter-token latency against batch size to find where it bends.
- Check KV cache usage: when it fills, the scheduler preempts sequences and swaps or recomputes them, which shows up as long stalls for exactly those requests.
- Check whether long prompt prefills share steps with decodes and stall them; chunked prefill or a cap on batched tokens per step smooths this, and queueing time adds to the tail.
What determines time to first token versus time per output token, and how would you improve each?
- Time to first token is queueing plus prefill, one compute-bound pass over the whole prompt, so it grows with prompt length and with load on the server.
- Time per output token is one decode step, bound by reading the weights and KV cache from memory, so it rises with model size, context length and batch size.
- Prefix caching, shorter prompts and separate prefill capacity help the first; quantization, speculative decoding, tensor parallelism and smaller batches help the second.
What to study for Batching, continuous batching and LLM serving
How do multi-query and grouped-query attention shrink the KV cache, and how would you implement GQA?
- KV cache size is 2 × layers × n_kv_heads × head_dim × sequence length × batch × bytes per element, so cutting the number of K/V heads shrinks it proportionally.
- Multi-head attention keeps one K/V head per query head, multi-query shares a single K/V head across all of them, and grouped-query sits between, with each group of query heads sharing one.
- To implement GQA, project K and V to n_kv_heads, repeat each across its n_heads/n_kv_heads query heads, then run normal attention; quality stays close to multi-head while decoding reads far fewer bytes.
FlashAttention computes exact attention yet runs faster than the standard version; where does the speedup come from?
- Standard attention writes the n by n score and softmax matrices to GPU high-bandwidth memory and reads them back, so it is limited by memory traffic rather than arithmetic.
- FlashAttention processes Q, K and V in tiles held in on-chip SRAM, using an online softmax that keeps a running max and sum and rescales partial outputs as each new block arrives.
- The full score matrix never reaches HBM, so extra memory is linear in sequence length, and the backward pass recomputes scores from saved softmax statistics; FLOPs stay O(n²), only data movement falls.
Mixture of experts
Asked often
How does top-k routing work in a mixture-of-experts layer, and why does it need load balancing?
- A small router computes a softmax over experts for each token, sends the token to its k highest-scoring experts, and outputs the gate-weighted sum of their outputs.
- Left alone the router collapses: a few experts get most tokens, improve faster and attract even more, so the rest stay undertrained and the busy ones overflow.
- An auxiliary loss rewards spreading tokens evenly across experts, and each expert has a capacity limit, so tokens beyond it skip that layer’s experts and are carried forward by the residual connection.
An MoE model has far more total parameters than active ones; what does that mean for training and serving?
- Compute per token tracks the active parameters, so Mixtral 8x7B, with about 47B total and 13B active, runs at roughly the FLOPs of a 13B dense model.
- Memory tracks the total, since every expert must stay loaded, so hosting costs resemble a dense model of full size, and a large batch touches most experts each step.
- Expert parallelism spreads experts across GPUs, but every MoE layer then needs all-to-all communication to send tokens to their experts and back, and imbalanced routing leaves some GPUs idle.
Would a 70B model fit on two 80 GB GPUs for inference? Walk me through the estimate.
- In bf16 the weights take 70B × 2 bytes, about 140 GB, against 160 GB of total memory, leaving roughly 20 GB before activations and framework overhead.
- With 80 layers and 8 KV heads of dimension 128, the KV cache costs 2 × 80 × 8 × 128 × 2 bytes, about 0.33 MB per token, so tens of thousands of tokens fit.
- Split the weights across both GPUs with tensor parallelism; int8 would halve them to about 70 GB, freeing far more room for KV cache and larger batches at a small quality cost.
Where does the rule of about 2N FLOPs per token come from, and when does it break down?
- Multiplying an (m × k) matrix by a (k × n) one costs 2mnk FLOPs, a multiply and an add each, so every weight contributes about 2 FLOPs per token in a forward pass.
- Training costs about 6N per token, because the backward pass computes gradients for both activations and weights and so costs roughly twice the forward pass.
- It ignores attention scores, about 4 × layers × context length × d_model FLOPs for a token attending over its context, so very long contexts make attention a meaningful share of compute.
Reasoning models and test-time compute
Asked often
What’s the difference between an outcome reward model and a process reward model, and how is each used?
- An outcome reward model scores only the final answer or whole solution, while a process reward model scores each reasoning step, giving finer credit for where a solution went wrong.
- At inference either can rerank best-of-N samples, and a process model can also guide search, such as beam search that keeps the most promising partial solutions.
- Step labels are expensive to collect and learned rewards can be gamed, which is why R1-style training relied on rule-based rewards from checkable answers instead.
Your team wants to switch a product feature to a reasoning model; how would you decide?
- Reasoning models pay off on multi-step problems with checkable answers, such as maths, code or planning, and add little to lookup, extraction, classification or short chat.
- Thinking tokens are usually billed as output and generated before the answer, so they raise cost and the time to the first visible token, which matters for interactive features.
- Compare on your own eval set, measuring accuracy against tokens and latency, and consider routing only hard requests to the reasoning model or lowering its reasoning effort.
How does GRPO differ from PPO, and why does it pair well with verifiable rewards?
- PPO trains a value model, usually similar in size to the policy, to estimate a baseline for each token, then updates with a clipped probability ratio and a KL penalty.
- GRPO drops the critic: it samples a group of answers per prompt and uses each answer’s reward minus the group mean, divided by the group standard deviation, as its advantage.
- Verifiable rewards like passing unit tests or a correct final answer are cheap and hard to game, but exist only for checkable tasks, and groups that all pass or all fail give no signal.
Alignment: instruction tuning, RLHF and DPO
Specialist topic or differentiator
Walk me through RLHF. What are the reward model and the KL penalty for?
- Supervised fine-tuning on demonstrations first gives an instruction-following model, then a reward model is trained on human rankings to score a response for a given prompt.
- The reward model uses a pairwise loss, −log σ(r(x, y_w) − r(x, y_l)), because people compare two responses more consistently than they score one on an absolute scale.
- PPO then maximises reward minus β times the KL divergence from the frozen SFT model, which stops the policy drifting into text that games the reward model or loses fluency.
What is the DPO loss, and when would you choose DPO over PPO-based RLHF?
- DPO minimises −log σ(β[log π_θ(y_w|x)/π_ref(y_w|x) − log π_θ(y_l|x)/π_ref(y_l|x)]), raising the chosen response’s likelihood relative to a frozen reference model and lowering the rejected one’s.
- It works because the optimal KL-constrained policy implies a reward of β log(π/π_ref) plus a prompt-only term, which cancels in pairwise comparisons, so no reward model is needed.
- Choose DPO for simplicity and memory, since it needs only the policy and a reference with no sampling loop; PPO’s online sampling can explore beyond a fixed preference set.
Speculative decoding and other inference accelerations
Specialist topic or differentiator
How does speculative decoding verify drafted tokens without changing the target model’s output distribution?
- A cheap draft proposes several tokens with probabilities q, and one forward pass of the target over all of them gives its own probabilities p at every position.
- Each drafted token x is accepted in order with probability min(1, p(x)/q(x)); at the first rejection, a replacement is sampled from max(0, p − q) renormalised and later drafts are discarded.
- This rule provably reproduces sampling from p, and if every draft is accepted the same target pass supplies one bonus token, so each iteration yields at least one token.
What controls the speedup from speculative decoding, and when does it stop helping?
- With acceptance rate α and γ drafted tokens, each target pass yields (1 − α^(γ+1)) / (1 − α) tokens on average, so speedup depends on α, γ and the draft’s relative cost.
- Longer drafts help only while acceptance stays high, since each extra token is less likely to survive yet still costs draft time, so γ is tuned or adapted to acceptance.
- Gains shrink at large batch sizes, where the target is already compute-bound and verifying extra tokens isn’t free, and on high-temperature or unusual text where drafts are rarely accepted.
What to study for Speculative decoding and other inference accelerations
Why inference is memory-bound: GPU basics for LLM engineers
Specialist topic or differentiator
Using arithmetic intensity, explain why prefill is compute-bound and decode is memory-bandwidth-bound.
- Arithmetic intensity is FLOPs per byte moved; a chip’s ridge point is peak FLOPs divided by memory bandwidth, and work below it waits on memory rather than arithmetic.
- Each bf16 weight is 2 bytes and does 2 FLOPs per token, so intensity is roughly the number of tokens processed together: the prompt length in prefill, the batch size in decode.
- Batching raises decode intensity for the weight matmuls, but attention reads each sequence’s own KV cache, which isn’t shared across the batch, so that part stays bandwidth-bound.
Estimate the best-case decode speed for a 13B model at batch size 1 on a GPU with 2 TB/s of memory bandwidth.
- Each decode step must read every weight once: 13B parameters × 2 bytes in fp16 is 26 GB, so one step takes at least about 13 ms.
- That caps generation at under 80 tokens per second; real systems land below it, since achieved bandwidth falls short of the peak and the KV cache adds reads.
- Batching barely changes that step time while decode is memory-bound, so throughput scales almost linearly with batch, and int8 weights halve the bytes read, roughly doubling the ceiling.
Long context: RoPE scaling, sliding windows and lost in the middle
Specialist topic or differentiator
How would you extend a model trained at a short context to a much longer one, and would you trust the result?
- Beyond its trained length RoPE produces rotation angles the model never saw, so position interpolation scales positions down into the trained range, followed by brief fine-tuning on long documents.
- NTK-aware scaling instead raises the RoPE base so high-frequency dimensions barely change, and YaRN interpolates each frequency band differently and adjusts the attention temperature.
- Passing needle-in-a-haystack is not enough, since recall drops for facts in the middle and with distractors, so test on realistic long tasks and compare against retrieval.
Small models and on-device inference
Specialist topic or differentiator
How would you design an on-device assistant, and what would you send to the cloud?
- Start from the memory budget: a 3B model at 4 bits is about 1.5 GB of weights, plus KV cache, sharing RAM with the operating system and other apps.
- Quantize aggressively and measure the quality cost on your own tasks, then share one base model across features with small task-specific LoRA adapters swapped in at runtime.
- Route requests needing broad knowledge, long context or harder reasoning to a server model, and measure latency, memory and battery on the target device, not a GPU box.
More LLM topics
What interviewers check on each, and what to study, in order.
Questions and answer points are written for SAADHO from each topic’s sources: question types interviewers ask, never leaked questions. How SAADHO is made