GenAI system design
- System design
- Asked in most interview loops
- About 2 h to learn
- 3 vetted resources
What this topic covers
Designing an LLM product end to end: how a request flows from the user through context construction, guardrails, model routing, caching and tool calls, and how you watch it in production. Interviewers give an open prompt (a support bot, a document Q&A assistant, an agent that takes actions) and expect you to start from requirements and a minimal pipeline, then add components only when a stated problem calls for them. Strong answers put numbers on latency, cost and quality, say where failures get caught (input and output checks, retries, human fallback) and explain how logs, traces and user feedback feed the next iteration. Common follow-ups: when to cache semantically, when a router or gateway pays for itself, and how to keep write actions safe from prompt injection. Strong answers also ask whether the feature needs an LLM at all, and keep retrieval inside each user's access rights.
You’re ready when you can
What interviewers check, from the resources below.
- Draw the five-step architecture from memory and say what problem each added component solves
- Name three output failure types and how you'd detect each one
- Compare prompt, exact and semantic caching and state when semantic caching does more harm than good
- Explain what a model gateway centralises and why a router can cut cost
- List the metrics and traces you'd log on day one of a launch
- Sketch the three stages of the in-context learning pattern and the components in each
- Explain why the authors favour retrieval over fine-tuning or very long context windows for most apps
- Name the operational pieces (logging, validation, caching) that sit around the model call
- Describe the routing → retrieval → generation pipeline LinkedIn shipped
- Explain how they made structured output from the LLM reliable when calling internal APIs
What to study, in order
Each link opened and checked by hand; the date says when.
Start here
Building A Generative AI Platform (opens in a new tab)
huyenchip.com · Chip Huyen · 50 min · Free · checked 21 Oct 2026
- Why
- It walks the exact shape of a GenAI design answer: start with a bare model call and add one component at a time, each justified by a problem it solves.
- What to read
- Read “Step 1. Enhance Context” through “Traces” (Steps 1–5 plus “Observability”), skimming “RAGs” since you've covered retrieval already.
- What to skip
- “AI Pipeline Orchestration” and the tool lists in “Gateway”; the references section.
Free alternative
Emerging Architectures for LLM Applications (opens in a new tab)
Andreessen Horowitz · Matt Bornstein, Rajko Radovanovic · 25 min · Free · checked 21 Oct 2026
- Why
- A compact reference diagram of the LLM app stack that gives you a vocabulary for naming the boxes on a whiteboard.
- What to read
- Read “The stack” and “Design pattern: In-context learning” (the three stages: preprocessing/embedding, prompt construction/retrieval, prompt execution).
- What to skip
- The vendor names in the tool lists, which date from June 2023, and “What about agents?”, which is out of date.
Go deeper
Musings on Building a Generative AI Product (opens in a new tab)
LinkedIn Engineering · Juan Pablo Bottaro, Karthik R. · 25 min · Free · checked 21 Oct 2026
- Why
- A real production case study with numbers, which is the kind of evidence that makes a design answer sound lived-in rather than textbook.
- What to read
- Read “Overview”, “What came easy” and “Where we struggled” (all four parts: evaluation, calling internal APIs, consistent quality, capacity & latency).
- What to skip
- “Takeaways”, which is mostly a product plug and thanks.
Companies that test System design most
Put GenAI system design in your plan
Tell us your interview date and hours a day; it lands on the right day, in the right order.