Cost and latency engineering for LLM apps
- System design
- Asked in most interview loops
- About 1 h 30 min to learn
- 3 vetted resources
What this topic covers
Once a prototype works, the next question is what it costs per request and how long the user waits. Both are dominated by token counts: inputs are priced per token and processed in parallel, outputs are generated one at a time and set the wall clock. The levers are shorter prompts and outputs, prompt caching of stable prefixes, routing easy requests to smaller models, batching or parallelising independent calls, streaming to cut perceived latency, and not calling an LLM at all when a rule or classifier will do. System-design rounds expect you to estimate cost per request, name TTFT vs. per-token latency, and explain which lever you would pull first and why.
You’re ready when you can
What interviewers check, from the resources below.
- List the seven latency principles from memory and give one concrete tactic for each
- Explain why cutting output tokens usually beats cutting input tokens
- Redesign the example customer-service bot to make fewer, parallel, shorter calls and estimate the saving
- Name two ways to lower perceived latency without lowering actual latency
- Compute cost per request and per 1,000 requests from input/output token counts and a price sheet
- Explain why input length affects latency far less than output length
- Compare prompting, finetuning and distillation as ways to cut per-call cost
- Explain why you should not hard-code a cost or latency budget around one vendor's current pricing
- Explain prefix matching and why only the stable prefix of a prompt should be cached
- State the write and read price multipliers relative to base input price and compute break-even for a given hit rate
What to study, in order
Each link opened and checked by hand; the date says when.
Start here
Latency optimization (opens in a new tab)
OpenAI Developer Docs · OpenAI · 30 min · Free · checked 21 Oct 2026
- Why
- “Our LLM feature is too slow, what do you do?” is a staple system-design prompt, and this guide is a ready-made checklist of levers in priority order.
- What to read
- Read all seven principles, “Process tokens faster.” through “Don't default to an LLM.”, then work the “Example” including “Analysis and optimizations”.
- What to skip
- Nothing; the page is short. Treat vendor-specific feature names as illustrative.
Free alternative
Building LLM applications for production (opens in a new tab)
Chip Huyen's blog · Chip Huyen · 20 min · Free · checked 21 Oct 2026
- Why
- Shows the back-of-envelope cost-per-prediction and latency reasoning that interviewers expect you to do on a whiteboard, plus the prompt-vs-finetune-vs-distil trade-off.
- What to read
- Read “Cost and latency” (“Cost”, “Latency”, “Cost + latency analysis for LLMs”) and “Prompting vs. finetuning vs. alternatives”.
- What to skip
- Part 2 and Part 3; the dollar figures and model names are from 2023, so keep the method and ignore the numbers.
Go deeper
Prompt caching (opens in a new tab)
Claude Developer Platform docs · Anthropic · 35 min · Free · checked 21 Oct 2026
- Why
- Prompt caching is the single biggest cost lever for long-system-prompt and agentic apps, and candidates are expected to know what it costs, what it caches, and what silently breaks it.
- What to read
- Read “How prompt caching works”, “Pricing”, “Explicit cache breakpoints” (including “When to use multiple breakpoints”), and “Caching strategies and considerations” up to and including “What invalidates the cache” and “Best practices for effective caching”.
- What to skip
- “Supported models”, “1-hour cache duration”, “Pre-warming the cache” and the code examples unless you are building with the API this week.
Companies that test System design most
Put Cost and latency engineering for LLM apps in your plan
Tell us your interview date and hours a day; it lands on the right day, in the right order.