Skip to content

Long context: RoPE scaling, sliding windows and lost in the middle

  • Transformers
  • Specialist topic or differentiator
  • About 75 min to learn
  • 4 vetted resources
Build a plan that includes this

What this topic covers

Models are trained at one context length and then stretched to longer ones. Common tools are RoPE position interpolation, NTK-style base changes and YaRN, plus ALiBi biases, sliding-window attention and attention sinks to keep memory bounded. A longer window doesn’t mean the model uses all of it well: models recall facts in the middle of long inputs worse than facts at the start or end, and quality drops as inputs grow, especially with distractors. Interviewers ask because this decides real design choices, like when to use long context and when to use RAG. A strong answer explains one extension method, why quality degrades with length, and how you would handle input beyond the window: retrieval, chunked map-reduce summaries, or putting key facts at the edges.

Builds on

You’re ready when you can

What interviewers check, from the resources below.

  • Can explain why RoPE breaks past its trained length and how interpolation helps
  • Can say how YaRN differs from plain position interpolation
  • Can explain sliding-window attention’s O(n·w) cost and what it gives up
  • Can choose between long context and RAG for a given workload
  • Can explain why plain window attention collapses once the first tokens are evicted
  • Can describe keeping a few sink tokens plus a rolling window of recent tokens
  • Can explain why attention sinks give fluency, not longer recall
  • Can explain why passing needle-in-a-haystack doesn’t prove a model uses long context well
  • Can name two factors that make long-input performance drop, such as low needle-question similarity and distractors
  • Can propose mitigations: retrieve and trim, put key facts at the start or end, summarise history

What to study, in order

Each link opened and checked by hand; the date says when.

  1. Start here

    Long Context LLMs: RoPE Scaling, Retrieval, and the Path to 1M Tokens (opens in a new tab)

    Machine Learning at Scale · 30 min · Free · checked 21 Oct 2026

    Why
    “How do you take a model from 8K to 128K?” and “when would you use long context instead of RAG?” come up often. This post covers both in one sitting.
    What to read
    Read Why Context Windows Are Hard to Extend (The RoPE Problem), the four Technique sections (Positional Interpolation, YaRN, LongRoPE, Adjusting the RoPE Base), “Lost in the Middle”, Long Context vs. RAG and Efficient Attention for Long Contexts.
    What to skip
    Skim The Training Data Challenge and skip the course promotions around the article.

  2. Free alternative

    Attention Sinks in LLMs for endless fluency (opens in a new tab)

    Hugging Face (community blog) · Tom Aarsen · 20 min · Free · checked 21 Oct 2026

    Why
    Sliding windows with attention sinks are the usual answer to “how do you keep a chatbot fluent without memory growing forever?” This post shows the failure and the fix with perplexity plots.
    What to read
    Read Limitations for Chat-Assistant LLMs, Window Attention, Attention Sinks with its Perplexity Experiments, and the FAQ. The FAQ makes clear that sinks keep the model fluent but don’t widen what it can recall.
    What to skip
    Skip Attention Sinks in Practice, which documents a 2023 package, and the endless-generation and chat-assistant experiments.

  3. Go deeper

    Context Rot: How Increasing Input Tokens Impacts LLM Performance (opens in a new tab)

    Chroma Research · Kelly Hong, Anton Troynikov, Jeff Huber · 30 min · Free · checked 21 Oct 2026

    Why
    This report gives you evidence that a large window doesn’t mean reliable use of it. That backs up any argument for retrieval or context trimming.
    What to read
    Read the Introduction, Needle in a Haystack Extension, the Experiment and Results parts of Needle-Question Similarity and Impact of Distractors, LongMemEval, and Limitations & Future Work.
    What to skip
    Skip Repeated Words, Haystack Structure details and the Appendix.

  4. Go deeper

    Lost in the Middle: How Language Models Use Long Contexts (opens in a new tab)

    arXiv · Nelson F. Liu et al. · 30 min · Free · checked 22 Oct 2026

    Why
    The U-shaped curve, where models use information at the start and end of a long context but miss the middle, is the standard evidence in “long context or RAG?” discussions.
    What to read
    Read §2 with Figure 1 (multi-document QA as the gold document moves), §3 on the synthetic key–value retrieval task, and §4 on why it happens (architecture, query-aware placement, instruction tuning). Then read §5 on why more retrieved documents stop helping.
    What to skip
    Skip §6 related work and Appendix G’s extra document-count results.

Companies that test Transformers most

Put Long context in your plan

Tell us your interview date and hours a day; it lands on the right day, in the right order.

Build my plan