Skip to content

Model routing and cascades

  • System design
  • Asked often
  • About 75 min to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

Routing decides up front which model should answer a query, while a cascade tries a cheap model first and escalates to a stronger one only when the answer looks unreliable. Interviewers raise it when a design is too expensive or too slow, because sending every request to the biggest model is rarely necessary. A strong answer names the routing signals (a trained classifier, the cheap model’s confidence, a verifier or quality scorer), explains how you’d evaluate the router itself on a cost versus quality curve, and notes the hidden costs such as added latency on escalated calls. It also covers the operational side: fallback to another provider on outages or rate limits, and monitoring for drift in the routing mix.

Builds on

Leads to

You’re ready when you can

What interviewers check, from the resources below.

  • Can name the four router types and the training data they used
  • Reads a cost versus quality curve and explains a point like “95% quality at a fraction of strong-model calls”
  • Explains why a router trained on one data mix can fall to near-random on another
  • Can describe how you’d evaluate a router against a random-routing baseline
  • Can sketch a cascade with a scorer and a per-step threshold
  • Explains how the scorer is trained and what happens when it’s miscalibrated
  • Can say why cheap models sometimes beat the expensive one on specific queries
  • Can list routing signals: difficulty classifier, preference data, uncertainty, verifier
  • Knows router metrics such as area under the cost-quality curve and APGR
  • Can name router failure modes and how you’d monitor for them

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing (opens in a new tab)

    LMSYS · Isaac Ong et al. · 25 min · Free · checked 21 Oct 2026

    Why
    When an interviewer says “this costs too much”, routing between a strong and a weak model is a standard answer. This post shows how routers are trained and, just as useful, how they’re judged on cost against quality.
    What to read
    Read “Routing Setup” and “Results”, including “RouteLLM vs Commercial Offerings” and “Generalizing to Other Models”. Note how cost is counted as the share of calls sent to the strong model, and why the MMLU router needed extra golden labels.
    What to skip
    Skip “Acknowledgements” and “Citations”; the linked paper is optional.

  2. Free alternative

    FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (opens in a new tab)

    arXiv · Lingjiao Chen, Matei Zaharia, James Zou · 45 min · Free · checked 21 Oct 2026

    Why
    This is the paper people cite for cascades: try the cheapest model, score its answer, and only escalate when the score is low. It gives you the vocabulary for a cascade design.
    What to read
    Read section 3, “How to Use LLMs Affordably and Accurately”, focusing on “Strategy 3: LLM cascade”, then section 4’s “A Case Study” and “Performance and Cost Trade-offs”.
    What to skip
    Skip “Strategy 1: Prompt adaptation”, “Strategy 2: LLM approximation” and the detailed dataset setup.

  3. Go deeper

    Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey (opens in a new tab)

    arXiv · Yasmin Moslem, John D. Kelleher · 60 min · Free · checked 21 Oct 2026

    Why
    A recent survey that maps the whole field, useful when an interviewer asks what signals a router could use or how you’d know it’s working.
    What to read
    Read 1.2 “Routing and Cascading” and 1.4 “Conceptual Design Space for LLM Routing”, then section 6 “Uncertainty-based Routing”, section 7 “Cascades” and section 9 “Evaluation” (especially 9.2 metrics and 9.4 failure modes).
    What to skip
    Skip section 8 “Multimodal Model Routing” and the RL method details in section 5 unless the role needs them. Provider fallback for outages isn’t covered here; think it through yourself.

Companies that test System design most

Put Model routing and cascades in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan