Skip to content

RL for LLMs: PPO, GRPO and verifiable rewards

  • Transformers
  • Asked often
  • About 3 h to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

Reasoning models are trained with reinforcement learning, and since DeepSeek-R1 the standard recipe is GRPO with rewards a program can check, such as a passing unit test or a correct final answer. Interviewers now ask “what is GRPO and why did DeepSeek use it over PPO?” and “when does a verifiable reward beat a learned reward model?”, even for applied roles. A strong answer explains what PPO’s critic does and how GRPO replaces it with a group-mean baseline, why the KL term to a reference model is there, where verifiable rewards work and where they don’t, and how reward hacking shows up and gets caught.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Explain what PPO’s critic, clipping and KL penalty each do
  • Explain how GRPO gets its baseline from a group of sampled answers and what that costs
  • Say when a verifiable reward beats a learned reward model, and where it can’t be used
  • Describe the DeepSeek-R1 training stages in order
  • Write the GRPO advantage for one prompt’s group of samples
  • Explain why a group where every answer is right gives no learning signal
  • Compare REINFORCE, RLOO, PPO and GRPO in one sentence each
  • Explain how per-token versus per-sequence loss averaging changes training
  • Explain how GRPO computes advantages without a value model
  • Describe the reward design and why rule-based rewards resist hacking

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    The State of Reinforcement Learning for LLM Reasoning (opens in a new tab)

    Ahead of AI · Sebastian Raschka · 60 min · Free · checked 22 Oct 2026

    Why
    “What is GRPO and why did DeepSeek-R1 use it over PPO?” is now a reported interview question. This one article walks from RLHF through PPO to GRPO and RLVR in the order an interviewer would probe.
    What to read
    Read “RLHF basics”, “A brief introduction to PPO”, “RL algorithms: from PPO to GRPO”, “RL reward modeling: from RLHF to RLVR” and “How the DeepSeek-R1 reasoning models were trained”. Then skim the six numbered lessons in “Lessons from recent RL papers”.
    What to skip
    Skip the 15 paper summaries in “Noteworthy research papers” on a first pass; come back to one or two if a role is post-training heavy.

  2. Free alternative

    Reinforcement Learning | RLHF and Post-Training Book (opens in a new tab)

    RLHF Book · Nathan Lambert · 75 min · Free · checked 22 Oct 2026

    Why
    A free, more rigorous treatment of the same algorithms, written by someone who trains open models. Useful when an interviewer asks you to write the GRPO advantage or compare it with RLOO.
    What to read
    Read “Policy Gradient Algorithms” from “Vanilla Policy Gradient” through “Group Relative Policy Optimization (GRPO)” and “Comparing Algorithms”. Then read “Implementation: Example: GRPO” and “Loss Aggregation Tradeoffs”.
    What to skip
    Skip the derivation in “Deriving the Policy Gradient” unless you need it, and skip GSPO, CISPO, “Asynchronous RL Systems” and “Truncated Importance Sampling” for applied roles.

  3. Go deeper

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (opens in a new tab)

    arXiv · DeepSeek-AI · 60 min · Free · checked 22 Oct 2026

    Why
    R1 showed that reinforcement learning with verifiable rewards alone can produce long chain-of-thought reasoning, and GRPO, R1-Zero and distillation into small models are now standard interview topics.
    What to read
    Read §2.1 (GRPO), §2.2 (rule-based accuracy and format rewards) and §2.3 on R1-Zero’s behaviour during training. Then §3 on the multi-stage R1 pipeline, and from the appendix A.3 (GRPO versus PPO), B.5 (reward hacking), F.1 (distillation versus RL) and G.2 (what didn’t work: PRMs and MCTS).
    What to skip
    Skip §4’s benchmark tables beyond the headline, §5 ethics, B.1 infrastructure, the D safety report and the per-category breakdowns in E.

Companies that test Transformers most

Put RL for LLMs in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan