I keep seeing the same debate pop up in AI engineering circles: "Is RAG dead now that context windows hit a million tokens?"
Why long context is tempting
The appeal isn't just size, it's that it lets you skip a whole pipeline. No chunking strategy. No embedding model. No vector database. No reranking logic. You just hand the model the source material and let it reason directly.
That also fixes something RAG has always struggled with: the "retrieval lottery." If your retrieval step misses the right chunk, the model never even gets a shot at the answer. Long context sidesteps that entirely, and it's genuinely better at questions that span an entire corpus, like spotting contradictions across 40 documents or finding what's missing between two reports. RAG hands you fragments; long context hands you the whole picture.
Why RAG isn't going anywhere, though
Three things keep it relevant:
The cost math doesn't disappear. RAG pays an indexing cost once and retrieves cheaply after that. Long context re-processes a huge chunk of text on every single query. At real production volume, that gap in cost and latency adds up fast.
More context doesn't mean better recall. Dumping a critical fact somewhere inside thousands of pages doesn't guarantee the model actually finds and uses it. RAG takes the opposite approach: don't make the model search the haystack, remove the haystack before it ever gets there.
And a million tokens sounds massive until you compare it to actual enterprise data. Most companies aren't sitting on one big document, they're sitting on millions of documents, tickets, emails, logs, and records that keep changing. At that scale you need retrieval, filtering, and routing.
So what should you actually build?
Long context makes sense when your dataset is bounded, you need reasoning across the whole thing, and simplicity beats optimization.
RAG makes sense when your knowledge base is large or growing, most queries only need a slice of it, and cost or latency actually matter.
And honestly, the best setups I'm seeing lately do both: retrieval narrows down what's relevant, then a large context window reasons over that narrowed set. It's a meaningfully different architecture than the RAG pipelines everyone was building a couple of years ago.
The real evolution isn't "RAG is dying." It's that context has become a first-class architectural resource, one more dimension to design around instead of a fixed constraint.
Curious what others are seeing in production, RAG, long context, or hybrid?
← Back to posts