The context window got big enough to make retrieval look optional

Context windows are now large enough that a reasonable engineer can ask why we still build retrieval systems. Claude Sonnet sits around a million tokens, Gemini 3 Pro around two million, Llama 4 Scout at ten. At those sizes, a lot of corpora fit.

The temptation is obvious: retrieval is infrastructure — chunking, embedding, an index, a re-ranker, an evaluation harness — and "put it all in the prompt" is one line of code.

Two numbers explain why that instinct is expensive.

Cost and accuracy both push the same way

RAG is roughly 1,250 times cheaper per query than stuffing a large corpus into context. That is not a tuning difference; it is a different order of magnitude. If your feature runs once a day for internal use, ignore it. If it runs per user request, it decides whether the product has a viable margin.

The second number is more surprising. Long context loses 30%+ accuracy when relevant content is buried mid-window. The model does not attend uniformly across a very long input; material in the middle gets systematically less weight. So the naive approach is not only more expensive, it can be less accurate — and it fails in a way that is hard to detect, because the answer is fluent and specific and simply wrong about the part you did not check.

Where each one genuinely wins

The research picture is reasonably consistent.

Long context is stronger when evidence is spread evenly across many documents and the question requires synthesis across them. Retrieval struggles here because there is no small set of chunks that contains the answer — the answer is the aggregate.

RAG is stronger when evidence is sparse: the answer lives in a few specific places, and the job is finding them. This describes most product features — documentation search, support answers, querying a knowledge base — where a handful of passages contain what is needed and the other 99% is noise that costs money and dilutes attention.

There is a failure mode on each side. RAG fails when retrieval misses, and it fails confidently, because the model answers from whatever it was given. Long context fails on burial and on cost.

The 2026 default is hybrid, and it is not a compromise

The pattern that has settled out is to retrieve to select and use a long window to reason: retrieval narrows a large corpus to a relevant subset, then a generous context lets the model reason across that subset without being forced into tiny top-k chunks.

This is better than either extreme for a reason worth stating. Early RAG systems were often limited to a handful of small chunks purely because context was scarce, which meant retrieval had to be nearly perfect. A large window relaxes that: you can retrieve fifty candidate passages instead of five, tolerate imperfect ranking, and let the model sort it out. Retrieval gets easier because the precision requirement dropped.

So the right way to read large context windows is not "retrieval is obsolete" but "retrieval no longer has to be exact".

Practical guidance

Instrument tokens per request before you argue about architecture. Most teams cannot state what a single user interaction costs them, which makes the entire discussion unfalsifiable.

If your corpus is small and static — a product manual, a policy document — long context alone is fine and the engineering saved is real. Do not build a retrieval pipeline for forty pages.

If your corpus is large, changing, or your feature runs per request, retrieve first. The cost gap is too large to argue with, and freshness is a genuine advantage: reindexing a document is cheap, and there is no equivalent to "the model already read it".

Put the most important material at the beginning or end of the context, not the middle. This is a real, measured effect and it is nearly free to act on.

Evaluate on retrieval quality separately from answer quality. When a hybrid system gives a wrong answer, you need to know whether the retriever missed or the model misread, and a single end-to-end score cannot tell you.

The pattern underneath

This is a recurring shape in engineering. A constraint relaxes, the workaround built around it starts to look unnecessary, and the reasonable-sounding move is to delete it.

Sometimes that is right. Here, the constraint that relaxed was context size, but the constraints that made retrieval valuable — cost per query, attention dilution, and the need to keep knowledge fresh — did not relax at all. The right response was not to remove retrieval but to stop asking it to be perfect.