Skip to content

Alignment: instruction tuning, RLHF and DPO

  • Transformers
  • Specialist topic or differentiator
  • About 1 h 40 min to learn
  • 6 vetted resources
Build a plan that includes this

What this topic covers

Alignment is the last stage of LLM training: a pretrained model is first instruction-tuned on demonstrations (SFT), then nudged toward answers humans prefer. Classic RLHF does this by training a separate reward model on ranked outputs and optimising the policy with PPO under a KL penalty that keeps it close to the SFT model. DPO collapses the reward model and the RL loop into one classification-style loss over preference pairs, which is why most open-weight chat models now use it or a variant. Interviewers ask you to draw the three-stage pipeline, explain what the reward model and KL term are for, write or at least describe the DPO loss, and argue when you would pick DPO over PPO (data, memory, stability).

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Describe the three training stages (pretraining, SFT, alignment) and what data each needs
  • Explain what the reward model is trained on and what the PPO step optimises
  • State why the policy is penalised for drifting from the SFT model (the KL term)
  • Name two ways Llama 2's RLHF differed from InstructGPT's
  • Draw the RLHF loop: policy, frozen reference model, reward model, and where the KL term enters
  • Explain why raters rank candidate outputs instead of scoring them on an absolute scale
  • Explain why the reward model and the policy are often different sizes
  • Write the DPO loss from memory and label the policy, the reference model and β
  • Explain the change of variables that lets the policy itself act as an implicit reward
  • Compare DPO with PPO-based RLHF on preference data needed, models held in memory and training stability

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    LLM Training: RLHF and Its Alternatives (opens in a new tab)

    Ahead of AI · Sebastian Raschka · 35 min · Free · checked 21 Oct 2026

    Why
    “How does a base model become a chat model?” comes up in nearly every applied-LLM loop, and this is the clearest map of the three stages and where RLHF and its replacements sit.
    What to read
    Read “The Canonical LLM Training Pipeline” through “RLHF in Llama 2” (including “Margin Loss”, “Two reward models” and “Rejection sampling”), then the DPO entry under “RLHF Alternatives”.
    What to skip
    The HIR, CPL and ReST entries under “RLHF Alternatives” and “Final Remarks”; they are a 2023 snapshot and rarely asked.

  2. Free alternative

    Illustrating Reinforcement Learning from Human Feedback (RLHF) (opens in a new tab)

    Hugging Face Blog · Nathan Lambert, Louis Castricato, Leandro von Werra, Alex Havrilla · 30 min · Free · checked 21 Oct 2026

    Why
    The explainer most interviewers learned RLHF from; it fixes the vocabulary (policy, reference model, reward model, KL penalty, PPO) you'll be expected to use at the whiteboard.
    What to read
    Read “Pretraining language models”, “Reward model training” and “Fine-tuning with RL”.
    What to skip
    “Open-source tools for RLHF” and “What's next for RLHF?”: the tooling and open questions are from late 2022 and have moved on.

  3. Go deeper

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model (opens in a new tab)

    arXiv · Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn · 45 min · Free · checked 21 Oct 2026

    Why
    “Explain DPO and why it needs no reward model” is the standard follow-up to any RLHF question, and Section 4 is the short derivation interviewers want you to reproduce.
    What to read
    Read “3 Preliminaries” and “4 Direct Preference Optimization”, then “5.1 Your Language Model Is Secretly a Reward Model”. Glance at “6.2 Can DPO scale to real preference datasets?” for the headline result.
    What to skip
    “5.2”, the rest of “6 Experiments”, “7 Discussion” and all appendices, unless you want to re-derive the loss yourself (then “A.2” and “A.4” only).

  4. Go deeper

    Training language models to follow instructions with human feedback (opens in a new tab)

    arXiv · Long Ouyang et al. · 50 min · Free · checked 22 Oct 2026

    Why
    InstructGPT is the reference recipe for SFT, then a reward model, then PPO, and most RLHF interview questions assume you know its three steps and its findings.
    What to read
    Read §3.1 with Figure 2 (the three-step method), §3.4 on how labellers were chosen and §3.5 on the SFT, reward model and PPO-ptx models. Then read §4.1 (a 1.3B aligned model preferred over 175B GPT-3) and §5.3 on limitations.
    What to skip
    Skip §4.2 public NLP results beyond the alignment-tax point, §5.5 broader impacts and most of Appendices A–E; look at C.2 and C.4 only if asked about training details.

  5. Go deeper

    Constitutional AI: Harmlessness from AI Feedback (opens in a new tab)

    arXiv · Yuntao Bai et al. · 45 min · Free · checked 22 Oct 2026

    Why
    Constitutional AI introduced RLAIF, replacing human harmlessness labels with AI feedback guided by written principles, and it comes up whenever an interviewer asks how to scale preference data.
    What to read
    Read §1.2 for the two-stage approach with Figure 1, §3.1 on the supervised critique-and-revise stage and §4.1 on the RL-from-AI-feedback stage. Then read §4.4 on harmlessness versus evasiveness, and skim Appendix C for the actual principles.
    What to skip
    Skip §2’s scaling plots, §3.4–3.5 and §4.5 unless you need detail, §5 related work and the prompt appendices D–E.

  6. Go deeper

    TRL - Transformers Reinforcement Learning (opens in a new tab)

    GitHub · huggingface · 60 min · Free · checked 22 Oct 2026

    Why
    TRL is the library most teams use for SFT, DPO and GRPO, and interviewers like candidates who can point to where the DPO loss and GRPO advantages are actually computed.
    What to read
    Read trl/trainer/dpo_trainer.py around _compute_loss and the “sigmoid” loss branch (the standard DPO loss). Then read trl/trainer/grpo_trainer.py where group rewards are turned into advantages (rewards minus the group mean, divided by the std). Look at trl/scripts/dpo.py for the full and LoRA command lines on Qwen2-0.5B-Instruct with trl-lib/ultrafeedback_binarized, and run the LoRA one on a free Colab GPU if you can.
    What to skip
    Skip the other loss types in the DPO trainer, vLLM generation and distributed options in GRPO, and the reward-model, online-DPO and KTO trainers.

Companies that test Transformers most

Put Alignment in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan