Lost in the Middle: Why Long-Running AI Agents Lose Instructions Buried in Their Own Context

Lost in the middle is the finding that a language model's ability to use a piece of information depends heavily on where that information sits in its input, not just on whether it's present. Put the relevant fact at the very start of the context or the very end, and the model finds it reliably. Bury the same fact in the middle, with the same total amount of surrounding text, and accuracy drops, sometimes below what the model scores with no supporting context at all. This is a positional effect, independent of context length: two inputs of identical size can produce very different answers depending only on where inside them the needed fact lives.

The effect was named and measured by Nelson Liu and coauthors in 2023, testing multi-document question answering and synthetic key-value retrieval across several models (Liu et al., "Lost in the Middle: How Language Models Use Long Contexts," arXiv:2307.03172). It matters to long-running agents for a specific structural reason: an agent's own turn-by-turn operation is what relocates its earliest instructions toward the middle. The original task brief does not move. Everything the agent does afterward, every tool call, every retrieved document, every intermediate result, accumulates after it, and each addition pushes that brief one step further from the "end" anchor where the model reads best. A session that runs long enough eventually buries its own starting instructions in exactly the zone the model is worst at using, with no error, no warning, and no change in context length required to explain why the agent stopped following them.

What the original study found

Liu et al.'s multi-document QA experiment is the clearest demonstration. With GPT-3.5-Turbo and 20 documents in context, accuracy was 75.8% when the document containing the answer was placed first, fell to 53.8% when placed in the middle of the stack, and recovered to 63.2% when placed last, a more than 20-point swing from a single change in position with the total amount of text held constant. The middle-position score, 53.8%, was lower than the model's closed-book score of 56.1%, meaning GPT-3.5-Turbo did worse with the correct answer present somewhere in its context than it did with no documents supplied at all. Extending the context window did not fix this: GPT-3.5-Turbo's 16K-context variant, tested with 30 documents, showed the same U-shape (73.4% first, 50.5% middle, 63.7% last). Claude-1.3 showed a shallower version of the same dip (59.9% first, 56.8% middle), and MPT-30B-Instruct and LongChat-13B showed it too, at different magnitudes. The pattern held across every model the paper tested; only the depth of the dip varied.

The paper's key-value retrieval task isolates the mechanism further by stripping away any need for reasoning: the model just has to return the value paired with a given key, out of a long list of key-value pairs, with the target pair's position in the list varied. Performance still sagged for keys in the middle of the list for several models, and in the worst case without any adjustment, accuracy fell to 45.6%. Placing the query itself both before and after the list of pairs, a technique the paper calls query-aware contextualization, restored near-perfect accuracy for that task. That is the single most concrete, tested fix in the literature for this specific failure mode, and it works because it removes the need for the model to hold the question in mind across the full distance to the buried value; the question is right next to the answer no matter where the answer sits.

Why a bigger context window doesn't make it go away

The intuitive fix, use a model with a larger advertised context window, does not hold up. A 2025 benchmark called NoLiMa tested 13 widely used models that each claim to support at least 128,000 tokens of context, specifically designing its test items so the question shares almost no literal wording with the passage containing the answer, which removes the lexical-overlap shortcut that makes many long-context benchmarks look better than real-world retrieval actually is (Modarressi et al., "NoLiMa: Long-Context Evaluation Beyond Literal Matching," arXiv:2502.05167). Every model performed well at short context lengths under 1,000 tokens. At 32,000 tokens, 11 of the 13 models fell below half of their own short-context baseline score. GPT-4o, one of the stronger performers in the set, dropped from a near-perfect 99.3% at short context to 69.7% at 32K. Giving a model a much larger window changes how much of the dip you can hide behind literal keyword matching; it does not change whether the model is actually finding and using a buried fact once that shortcut is removed.

This is the detail worth sitting with for anyone evaluating a model for a long-running agent deployment: a vendor's stated context window size tells you the input the model will accept, not how reliably it will use the part of that input sitting in the middle. The two numbers are frequently assumed to be the same thing, and the gap between them is exactly what both of these papers measured directly.

Why it compounds specifically in long-running agents

A single long document with a buried fact is a one-time instance of this problem. A long-running agent recreates the same structural setup on every turn, automatically, without anyone choosing to bury anything. The system prompt and original task brief are written once, at the start of the session, and then every subsequent tool call, retrieved file, and intermediate result gets appended after them. Nobody moved the original instructions; the agent's own continued operation is what carries the "end" anchor further away from them, turn by turn, until the brief is sitting deep in the part of the context the model reads worst, exactly when the session has accumulated the most reasons to still need it. This is one of the mechanisms behind behavioral state decay: a session that performed correctly for its first ten turns can start ignoring a constraint stated at turn one, not because the constraint was removed from context, and not because the model changed, but because the constraint's position relative to the growing transcript crossed into the zone where retrieval reliability is structurally weaker.

This is a distinct mechanism from context rot, which is the finding that accuracy on a fixed task degrades as total context volume grows, independent of where any particular fact sits. Lost-in-the-middle and context rot can both be operating on the same long-running agent at once, but they are separable: context rot shows up even when the relevant fact is at the end of the context, where lost-in-the-middle predicts the model should be strong; lost-in-the-middle shows up even at a fixed, moderate context length, where context rot predicts the model should be fine. It is also distinct from proactive interference, which degrades retrieval of the correct, most-recent value as a function of how many earlier, similar-looking values preceded it, not as a function of where in the context the correct value sits; the benchmark behind that effect deliberately places the current value last, closest to the query, specifically to rule out a positional explanation, and the decline still appears. And it is distinct from the recency-biased eviction described under context compaction failure modes: compaction failure is about content getting dropped from context entirely; lost-in-the-middle is about content that is still fully present and still being degraded in how reliably the model retrieves it.

Detecting it

The test that isolates this specific mechanism is a position-shuffle, not a length change: take a fixed set of context content and a fixed critical fact, and rerun the same task with that fact's position varied between the start, the middle, and the end, without changing the total token count at all. If accuracy swings meaningfully across those runs, the model's behavior is position-sensitive in a way a bigger or smaller context window alone won't fix. A cheaper proxy worth tracking in a live agent deployment: measure the token or turn distance between the original system prompt and the current point in the transcript, and flag sessions where that distance crosses a threshold you've established empirically degrades instruction-following, rather than assuming a long session is fine as long as it hasn't hit a hard context limit.

Mitigation

Query-aware contextualization is the one fix with direct experimental backing: restate the operative goal or constraint near the end of the context, immediately before the point where the model has to act, rather than relying on a single statement made once at the start of the session. This is the same underlying move as re-anchoring patterns, applied specifically to counter positional decay rather than general drift. Beyond restating, the more durable fix is to stop relying on transcript position at all for anything load-bearing: write the constraint or fact to an externalized, structured record that gets re-read at a fixed point in every turn, per state externalization, so its reliability doesn't depend on where it happens to have ended up inside an ever-growing transcript. Trimming the middle through summarization reduces how much content is competing inside the weak zone, but that approach inherits its own risks, covered separately under context compaction failure modes, so it is a complement to query-aware placement and externalization, not a substitute for either.

Related terms

• Context rot: the adjacent, volume-based degradation mechanism, distinct from this position-based one.
• Proactive interference: a different retrieval failure, driven by how many prior similar values preceded the target rather than by its position.
• Context compaction failure modes: what happens when content is dropped from context entirely, rather than degraded in place.
• Behavioral state decay: the umbrella phenomenon this mechanism feeds, once an early instruction becomes unreliable purely from its position in a growing session.
• Re-anchoring patterns: the mitigation pattern of restating goals mid-session, directly applicable to counteracting this effect.
• State externalization: the structural fix that removes load-bearing facts from transcript position entirely.

FAQ

What is "lost in the middle" in AI agents? It's the finding that a language model retrieves information least reliably when that information sits in the middle of its input context, versus the start or end, holding the total amount of context constant. In a long-running agent, the original task instructions drift toward that middle zone purely because later turns keep accumulating after them.

Is this the same as context rot? No. Context rot is accuracy declining as total context volume grows, regardless of where the relevant fact sits. Lost-in-the-middle appears even at a fixed, moderate context length; it is about position, not volume, and the two effects can compound in the same long session.

Does a larger context window fix it? No. The 2023 study found the same U-shaped pattern in a 16K-token variant of its test model as in the shorter-context version, and a 2025 benchmark testing 13 models that each claim 128,000-token-plus context found 11 of them fell below half their short-context accuracy at just 32,000 tokens once literal keyword overlap was removed from the test. A bigger advertised window changes how much of the problem a lexical shortcut can hide; it does not fix the underlying retrieval weakness.

What actually fixes it? The one technique with direct experimental support is query-aware contextualization: placing the question or operative instruction both before and after the content it needs to act on, rather than stating it once and letting new content accumulate after it. For a long-running agent, the more durable version of the same idea is externalizing load-bearing facts to a structured record re-read each turn, rather than depending on wherever those facts happen to land inside a growing transcript.

Does this affect every model equally? No. The original study found the effect at different depths across GPT-3.5-Turbo, Claude-1.3, MPT-30B-Instruct, and LongChat-13B, with Claude-1.3 showing a shallower dip than the others. The 2025 benchmark likewise found the size of the drop varied across the 13 models it tested, but found the underlying pattern, worse performance in the middle, in nearly all of them.