Skip to content

Mixture of experts

  • Transformers
  • Asked often
  • About 1 h 30 min to learn
  • 4 vetted resources
Build a plan that includes this

What this topic covers

A mixture-of-experts layer replaces one feed-forward block with many expert blocks and a small router that sends each token to its top-k experts. Interviewers ask about it because many leading open models are MoEs, and their costs behave differently from dense models. You should be able to explain top-k gating, why an auxiliary load-balancing loss is needed to stop the router collapsing onto a few experts, and how expert capacity limits mean overflow tokens get dropped or passed through. The key trade-off is that an MoE cuts FLOPs per token but still has to hold every expert in memory. A strong answer also covers serving: expert parallelism spreads experts across GPUs at the price of all-to-all communication.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Can describe top-k gating with a softmax router
  • Can explain why a load-balancing loss is needed and what router collapse looks like
  • Can explain capacity factor and what happens to overflow tokens
  • Can say an MoE cuts FLOPs per token but must still hold every expert in memory
  • Can explain expert parallelism and its all-to-all communication cost
  • Can sketch a sparse MoE layer with router, experts and weighted sum
  • Can explain the difference between active and total parameters using Mixtral 8x7B
  • Can describe how the Switch Transformer simplifies routing to top-1
  • Can write the auxiliary load-balancing loss in words
  • Can explain how capacity factor trades dropped tokens against wasted compute

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Mixture of Experts Explained (opens in a new tab)

    Hugging Face Blog · Omar Sanseviero, Lewis Tunstall, Philipp Schmid, Sourab Mangrulkar, Younes Belkada, Pedro Cuenca · 45 min · Free · checked 21 Oct 2026

    Why
    Many top open models are MoEs, so expect questions like “why is it cheap to run but expensive to host?” This post covers routing, balancing and serving in one place.
    What to read
    Read TL;DR, What is a Mixture of Experts (MoE)?, What is Sparsity?, Load balancing tokens for MoEs, Switch Transformers, Stabilizing training with router Z-loss, When to use sparse MoEs vs dense models?, and all of Making MoEs go brrr (Parallelism, Capacity Factor and communication costs, Serving techniques).
    What to skip
    Skip A Brief History of MoEs, Fine-tuning MoEs, Open Source MoEs and Exciting directions of work. The model lists are from 2023.

  2. Free alternative

    A Visual Guide to Mixture of Experts (MoE) (opens in a new tab)

    Exploring Language Models (Substack) · Maarten Grootendorst · 30 min · Free · checked 21 Oct 2026

    Why
    If the Hugging Face post feels dense, these diagrams make routing and capacity clear quickly. That helps when you’re asked to sketch an MoE layer on a whiteboard.
    What to read
    Read The Experts, The Routing Mechanism, Load Balancing (KeepTopK, Auxiliary Loss, Expert Capacity and the Switch Transformer subsections) and Active vs. Sparse Parameters with Mixtral 8x7B.
    What to skip
    Skip Mixture of Experts in Vision Models (Vision-MoE and From Sparse to Soft MoE).

  3. Go deeper

    Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (opens in a new tab)

    arXiv · William Fedus, Barret Zoph, Noam Shazeer · 60 min · Free · checked 21 Oct 2026

    Why
    Capacity factor, the auxiliary loss and expert parallelism all come from this paper. Reading it lets you answer follow-ups more precisely.
    What to read
    Read section 2 (Switch Transformer), especially 2.1 Simplifying Sparse Routing, 2.2 Efficient Sparse Routing and 2.4 Improved Training and Fine-Tuning Techniques. Then read 5.4 Expert and Data Parallelism.
    What to skip
    Skip section 4 Downstream Results and 5.6 Towards Trillion Parameter Models.

  4. Go deeper

    DeepSeek-V3 Technical Report (opens in a new tab)

    arXiv · DeepSeek-AI · 60 min · Free · checked 22 Oct 2026

    Why
    DeepSeek-V3 is the most-discussed modern MoE design, and interviewers use it for questions on fine-grained and shared experts, load balancing, multi-head latent attention and FP8 training.
    What to read
    Read §2.1.2 (DeepSeekMoE with shared and routed experts, and load balancing without an auxiliary loss) and §2.1.1 on multi-head latent attention, with Figure 2. Then §2.2 on multi-token prediction, the ablations in §4.5.1–4.5.3, and §3.4 on separate prefill and decode deployment.
    What to skip
    Skip §3.2 and §3.5 (pipeline and hardware details), most of §3.3 FP8 internals, §4.4 and §5.3 benchmark tables and the appendices.

Companies that test Transformers most

Put Mixture of experts in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan