Retrieval in 2026: Transitioning RAG from Encoders to LLMs

In 2023, the typical architecture for Retrieval-Augmented Generation (RAG) was fairly uniform, relying heavily on the BERT family (including BGE and e5) for semantic understanding, BM25 for exact matches, and cross-encoders for re-ranking, often with Qdrant as the underlying database. This setup has been the standard for two years, and many still utilize it. However, a closer look at the production environments of leading teams reveals a significant shift. The reliance on traditional encoders has diminished, with fine-tuned LLMs taking center stage. Re-ranking is now predominantly handled by LLMs as well, and inference is conducted using SGLang instead of ONNX. This transformation has necessitated a complete overhaul of the supporting infrastructure. This article delves into these changes and offers insights on how to adapt your technology stack, particularly if you are operating in a niche domain lacking ready-made datasets.