Skip to content

Distributed training: DDP, FSDP/ZeRO, tensor and pipeline parallelism

  • ML fundamentals
  • Specialist topic or differentiator
  • About 2 h to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

When one GPU is too slow or too small, training is split across many. Data parallelism (DDP) copies the model and all-reduces gradients every step. ZeRO and FSDP shard optimizer states, gradients and finally the weights themselves across those copies. Tensor parallelism splits individual matrix multiplies, and pipeline parallelism splits the layers. Interviewers ask “a 7B model won’t fit on one GPU, what now?” to see whether you can do the memory arithmetic (weights, gradients, Adam states, activations) and pick a strategy. A strong answer names the collective each method uses (all-reduce, all-gather, reduce-scatter), explains sharded versus replicated state, and says how 3D parallelism maps each kind onto the network hierarchy.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Estimates memory for a 7B model with Adam in mixed precision (about 16 bytes per parameter)
  • Says what DDP all-reduces each step and how it overlaps with backward
  • Explains ZeRO-1/2/3 and which collectives FSDP uses (all-gather, reduce-scatter)
  • Explains why tensor parallelism stays within a node and pipeline parallelism spans nodes
  • Explains the pipeline bubble and how micro-batches shrink it
  • Describes Megatron-style column and row splits for an MLP
  • Names three memory savers besides sharding and what each one trades away
  • Breaks the 16 bytes per parameter into weights, gradients and Adam states
  • Gives per-GPU memory for each ZeRO stage with N data-parallel ranks
  • Explains why ZeRO-3 adds about 50% communication over DDP

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    The Ultra-Scale Playbook: Training LLMs on GPU Clusters (opens in a new tab)

    Hugging Face (nanotron) · Nouamane Tazi, Ferdinand Mom, Haojun Zhao, Phuc Nguyen, Mohamed Mekkouri, Leandro von Werra, Thomas Wolf · 90 min · Free · checked 21 Oct 2026

    Why
    It builds from “what fits on one GPU” to full multi-axis parallelism, with memory and communication worked out at each step, which is how a strong interview answer is structured.
    What to read
    Read “First Steps: Training on One GPU” (memory for weights, gradients, optimizer states and activations; activation recomputation; gradient accumulation), all of “Data Parallelism” including the three ZeRO stages, “Tensor Parallelism” up to sequence parallelism, the first part of “Pipeline Parallelism”, and “5D Parallelism in a Nutshell”. Keep “A0: Parallel Programming Crash Course” open for the collectives.
    What to skip
    Skip “Context Parallelism”, “Expert Parallelism”, the later pipeline schedules (zero bubble, DualPipe), “Finding the Best Training Configuration” benchmarks and “Diving into the GPUs”.

  2. Free alternative

    How to Train Really Large Models on Many GPUs? (opens in a new tab)

    Lil'Log · Lilian Weng · 40 min · Free · checked 21 Oct 2026

    Why
    A shorter, well-organised tour of every parallelism type, useful when you have an evening rather than a weekend.
    What to read
    Read “Training Parallelism” (data, model, pipeline and tensor parallelism), then “Other Memory Saving Designs”: activation recomputation, mixed precision training and memory-efficient optimizers.
    What to skip
    Skip “Mixture-of-Experts” and the “Compression” and “CPU Offloading” subsections unless the role is about training infrastructure.

  3. Go deeper

    ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (opens in a new tab)

    arXiv · Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He · 45 min · Free · checked 21 Oct 2026

    Why
    FSDP is built on this design, and the paper’s memory accounting is the clearest source for the “where does the memory go?” follow-up.
    What to read
    Read the sections on where memory goes (model states versus residual states) and on ZeRO-DP’s three stages, including the communication-volume analysis showing stages 1 and 2 cost the same as plain data parallelism.
    What to skip
    Skip the ZeRO-R details (activation partitioning, buffers, fragmentation) and the large-scale experiment tables.

Companies that test ML fundamentals most

Put Distributed training in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan