Skip to content

LLM observability and feedback loops

  • Evals
  • Asked often
  • About 60 min to learn
  • 3 vetted resources
Build a plan that includes this

What this topic covers

Observability is how you find out what an LLM feature actually does once real users hit it: every request becomes a trace made of nested spans (LLM call, retrieval, tool call), each carrying inputs, outputs, latency, token counts and cost. On top of that sit online scores (LLM-judge or heuristic checks on live traffic), user feedback signals, and dashboards or alerts for cost, error rate and quality drift. The loop closes when bad traces are pulled out of production and added to the offline eval set. Interviewers ask this in system-design and 'tell me about a production incident' rounds: what you log, how you spot a hallucination spike without a ground truth, how you roll out a prompt change safely (shadow or canary), and how feedback gets back into evals.

Builds on

Leads to

You’re ready when you can

What interviewers check, from the resources below.

  • Define trace, observation (span / generation) and session, and draw how they nest for one RAG chat turn
  • Explain why tracing SDKs batch and send in the background, and what goes wrong in a serverless function if you forget to flush
  • Name three attributes you would attach to every trace (user, release, environment, tags) and say which dashboard question each answers
  • State the difference between tracing and observability in your own words
  • List the attributes every inference span should carry (operation name, request model, input/output tokens, finish reasons) and explain how the span name is built
  • Explain why prompt and completion content is treated as sensitive and give two places it can be stored instead of on the span
  • Sketch the span tree for an agent turn: invoke agent, inference, execute tool, retrieval
  • Say which of these attributes you would feed a cost dashboard and how
  • Explain why unit tests and staging environments are not enough for an LLM feature, with one concrete example of input drift
  • Describe what a wide, high-cardinality event for one LLM request should contain and how you would group by input or outcome

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Observability Data Model (Concepts) (opens in a new tab)

    Langfuse Docs · Langfuse · 20 min · Free · checked 21 Oct 2026

    Why
    Interviewers expect you to describe an LLM request as a trace of observations with sessions, users and tags on top, and this page gives that vocabulary in ten minutes.
    What to read
    Read “Core Concepts” through “Adding Attributes”, then “Built on OpenTelemetry” and “Background Processing” (including “Long-running applications” and “Short-lived applications”).
    What to skip
    “Instrumentation” SDK details and the “Agent Instructions” block at the end; they are tool-specific.

  2. Free alternative

    Semantic conventions for generative client AI spans (opens in a new tab)

    OpenTelemetry GenAI Semantic Conventions · OpenTelemetry · 30 min · Free · checked 21 Oct 2026

    Why
    Knowing the vendor-neutral span names and gen_ai.* attributes lets you answer 'how would you instrument this without locking into one tool' and shows you understand what a span must carry.
    What to read
    Read “Spans” through “Execute tool span” (Inference, Embeddings, Retrievals, Memory), then “Capturing instructions, inputs, and outputs” with its “Recording content on attributes” and “Uploading content to external storage” subsections.
    What to skip
    “Streaming chunks” and the provider-specific files (openai.md, anthropic.md, aws-bedrock.md); the spec is still marked Development, so learn the shape rather than memorising attribute names.

  3. Go deeper

    LLMs Demand Observability-Driven Development (opens in a new tab)

    Honeycomb Blog · Charity Majors · 15 min · Free · checked 21 Oct 2026

    Why
    This is the argument behind the tooling: you cannot test a nondeterministic model to completion, so production traces, user signals and quality scores become the dev loop, which is exactly the 'how do you know it works in prod' question.
    What to read
    Read “LLMs are their own beast”, “Observability-driven development is necessary with LLMs” and “So, what do you need to get started with LLMs?”.
    What to skip
    “100% debuggable? Maybe not” (framing) and the closing two sections, which are pep talk.

Companies that test Evals most

Put LLM observability and feedback loops in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan