Skip to content

LLM gateways: rate limits, token budgets, retries and idempotency

  • System design
  • Asked often
  • About 2 h to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

Every production LLM app sits behind provider limits measured in requests and tokens per minute, and most teams add their own gateway to share that quota fairly across users, retry safely and fail over to another model. It shows up both as a coding prompt (build a token-bucket limiter) and as a system-design follow-up (“what happens when you hit 429s at peak?”). A strong answer explains token-bucket versus fixed-window limiting, why LLM limits must count tokens and not just requests, per-tenant quotas and load shedding, backoff with jitter that respects retry-after, idempotency keys so a retried tool call or charge doesn’t run twice, and when to fall back to another provider.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Implement a token bucket with capacity and refill rate
  • Explain the difference between a rate limiter and a load shedder
  • Explain why a concurrency limit suits long LLM streams better than requests per second
  • Describe how you’d share one bucket across many workers
  • Explain why input and output tokens are metered separately
  • Explain how prompt caching raises effective throughput under the limit
  • Use retry-after and remaining-token headers to pace a client
  • Tell a retryable 429 apart from one that will keep failing
  • Explain the three points where a request can fail and which are safe to retry
  • Design an idempotency key for an agent’s “send refund” tool

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Scaling your API with rate limiters (opens in a new tab)

    Stripe · Paul Tarjan · 30 min · Free · checked 22 Oct 2026

    Why
    The classic write-up on how a real API protects itself with token buckets, concurrency caps and load shedding. Those are the same building blocks an LLM gateway needs, and the post gives you the vocabulary for the design round.
    What to read
    Read “Rate limiters and load shedders”, all four limiter types under “Using different kinds of rate limiters in concert”, and “Building rate limiters in practice”. Then sketch how you’d change the request limiter to count tokens instead of requests.
    What to skip
    Nothing major to skip; the linked gist with Redis scripts is optional.

  2. Free alternative

    Rate limits (opens in a new tab)

    Claude Docs · Anthropic · 20 min · Free · checked 22 Oct 2026

    Why
    Shows how a real LLM provider meters you: a token bucket over requests, input tokens and output tokens per minute, with cached tokens treated differently. Good for grounding your gateway design in real limits.
    What to read
    Read “About rate limits”, the “Rate limits” section including “Cache-aware ITPM”, “Setting lower limits for workspaces” and “Response headers”. Note how spend-cap 429s differ from rate-limit 429s.
    What to skip
    Skip the per-tier tables, Message Batches, Managed Agents, Files API and fast-mode sections.

  3. Go deeper

    Designing robust and predictable APIs with idempotency (opens in a new tab)

    Stripe · Brandur Leach · 25 min · Free · checked 22 Oct 2026

    Why
    Retries are only safe when repeating a call can’t do the work twice. This is the standard explanation of idempotency keys, backoff and jitter, and it applies directly to agents whose tool calls send emails or move money.
    What to read
    Read the whole post: “Planning for failure”, “Making liberal use of idempotency”, “Guaranteeing ‘exactly once’ semantics” and “Being a good distributed citizen”. Then map each failure case onto an agent retrying a tool call.
    What to skip
    Skip the closing “Codifying the design” section if short on time.

Companies that test System design most

Put LLM gateways in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan