Calling LLM APIs in Python: async, streaming, retries
- Coding
- Asked in most interview loops
- About 1 h 40 min to learn
- 3 vetted resources
What this topic covers
LLM applications are almost entirely I/O-bound: every model call, tool call and retrieval is a network round trip that takes hundreds of milliseconds to seconds. The practical coding round therefore often asks you to fan out many requests concurrently, cap in-flight calls to respect a rate limit, add timeouts and cancellation, and stream tokens to a client as they arrive. Interviewers look for whether you understand what asyncio actually does (cooperative scheduling on one thread), when it beats threads, how a blocking call silently kills throughput, and how to consume a server-sent-event stream from a provider SDK and forward it. Production questions add what happens when a call fails: retries with exponential backoff and jitter on 429s and 5xx errors, idempotency so a retry never does the work twice, and falling back to another model when one is down. This topic covers coroutines, tasks and TaskGroups, gather vs. structured concurrency, timeouts, Semaphore-based rate limiting, retries and fallbacks, and provider streaming.
You’re ready when you can
What interviewers check, from the resources below.
- Explain why calling a coroutine function does nothing until it is awaited or wrapped in a task
- Implement fan-out over a list of prompts with asyncio.TaskGroup and again with gather, and state how each behaves when one call raises
- Wrap a slow call in asyncio.timeout and catch the TimeoutError in the right place
- Explain why a plain blocking call inside a coroutine stalls every other task, and move it off the loop with asyncio.to_thread
- State why fire-and-forget tasks need a strong reference
- Explain concurrency vs parallelism in one minute using the I/O-wait argument, and say which one an LLM gateway needs
- Decide, for a given library call, whether the endpoint should be async def or def, and justify it
- Explain where a plain def endpoint runs in FastAPI and why that prevents it from blocking the server
- Name the case where a synchronous DB or model call inside an async def handler hurts throughput
- List the event sequence of a streamed message from message_start to message_stop and what each content_block event carries
What to study, in order
Each link opened and checked by hand; the date says when.
Start here
Coroutines and tasks (opens in a new tab)
Python documentation · Python Software Foundation · 40 min · Free · checked 21 Oct 2026
- Why
- “Call the model for these 500 prompts concurrently with a timeout” is a standard LLM-engineer coding prompt, and it is solved with exactly the primitives on this page.
- What to read
- Read “Coroutines”, “Awaitables”, “Creating tasks”, “Task groups”, “Running tasks concurrently”, “Timeouts” and “Running in threads”.
- What to skip
- “Eager task factory”, “Shielding from cancellation”, “Scheduling from other threads”, “Introspection” and the “Task object” reference.
Free alternative
Concurrency and async / await (opens in a new tab)
FastAPI documentation · Sebastián Ramírez (tiangolo) · 25 min · Free · checked 21 Oct 2026
- Why
- Most LLM services are FastAPI endpoints, and interviewers ask the one question this page is built around: when should a handler be async def, and what happens if you block inside it.
- What to read
- Read “In a hurry?”, “Asynchronous Code” (the burger analogy through “Concurrency + Parallelism: Web + Machine Learning”), “async and await” and “Coroutines”.
- What to skip
- “Very Technical Details” unless you are asked about dependency threadpools.
Go deeper
Streaming messages (opens in a new tab)
Claude Developer Platform docs · Anthropic · 35 min · Free · checked 21 Oct 2026
- Why
- Streaming is what makes a chat product feel fast, and candidates are asked to show they can consume an SSE event stream, reassemble it, and forward it, not just print tokens.
- What to read
- Read “Streaming with SDKs”, “Get the final message without handling events”, “Event types”, “Content block delta types”, then the “Basic streaming request” and “Streaming request with tool use” examples under “Full HTTP stream response”, and finish with “Error recovery”.
- What to skip
- The thinking and web-search streaming examples, and every non-Python SDK snippet.
Companies that test Coding most
Put Calling LLM APIs in Python in your plan
Tell us your interview date and hours a day; it lands on the right day, in the right order.