Retrieval Augmented Generation RAG

RAG is the most common form of grounding. Grounding is the overarching principle of linking model responses to verifiable external sources rather than solely to training patterns.

This way, current information or company-specific knowledge can be integrated without the model needing to be retrained.

Retrieval Augmented Generation is primarily used in enterprise AI for knowledge management, assistance for service and sales, technical documentation, and support chatbots.

A RAG pipeline retrieves relevant text passages for each user request from a document collection, a database, or a knowledge graph. It adds them to the prompt and has the language model generate a response with source attribution based on them.

The term was coined in 2020 by Patrick Lewis et al. in „Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks“ (arXiv 2005.11401, NeurIPS 2020).

Core idea: a parametric memory (the pretrained language model) is connected with a non-parametric memory (a dense vector index over Wikipedia). A neural retriever accesses it at inference time.

How RAG Works: The Four Steps of a Pipeline

A RAG pipeline has four phases: knowledge base indexing, retrieval at query time, prompt augmentation, and response generation. Three of these run with each individual query. Only indexing happens upfront or in the background.

Knowledge Base Indexing

In the indexing phase, source documents—manuals, tickets, product data, PDFs, wiki pages—are broken down into smaller sections (chunks). An embedding model converts them into vectors, which are stored in a vector database. Chunk sizes typically range from 128 to 512 tokens. Fact-based content with keyword matching benefits from smaller chunks (128–256 tokens), while conceptual content and summaries benefit from larger ones (256–512 tokens).

Dedicated vector databases (Qdrant, FAISS) or extensions of existing systems (pgvector for PostgreSQL, Elasticsearch/OpenSearch) are suitable as storage. Which choice is appropriate depends more on scaling and existing infrastructure than on the RAG concept itself.

Retrieval at query time

When a user question arrives, it is also embedded and matched against the vector index. The retriever provides the k most similar chunks. In practice, pure dense retrieval is rarely sufficient: exact tokens such as product codes, function names, or rare proper nouns are more reliably found by BM25. Dense vectors, on the other hand, capture paraphrases and intent. Therefore, hybrid search dominates in productive systems: both methods run in parallel, and then the result lists are merged.

Augmentation and Generation

The selected chunks are combined with the original question and a system instruction to form a prompt – this is the augmentation step. The LLM generates an answer from this and should cite the sources used. Only this reference to sources distinguishes a RAG answer from a free LLM answer: it makes the statement verifiable and shifts responsibility from „the model claims“ to „the model cites.“.

RAG, Fine-Tuning, or Long-Context? When to Use What

The three approaches are often presented as competitors, but they are answers to different questions. RAG does not change the model, but rather augments it with a knowledge source. Fine-tuning changes the weights. Long-context merely enlarges the input window.

RAG vs. Fine-Tuning

RAG is suitable when relevant knowledge changes frequently, when each answer must cite a source, or when the knowledge base needs to remain separate from the model for legal or organizational reasons. Content can be updated by re-indexing the vector database — without model training.

Fine-tuning is suitable when not knowledge, but behavior is to be adapted: tonality, response structure, a specific classification style. It is more expensive because it requires specialized labeling and training compute time. Once completed, the model remains with the knowledge state at that time. In practice, the approaches are not mutually exclusive—a domain-fine-tuned model as a generator in a RAG pipeline is a common combination.

RAG vs. Long Context Prompting

With context windows of several hundred thousand tokens, the theory emerged that RAG would become obsolete. This has not been confirmed. Anyone who shoves an entire manual into the prompt pays the full token price for every request. They risk quality losses due to diluted context (the documented lost-in-the-middle effect) and forgo the opportunity to selectively choose only the relevant passages. Long-context and RAG complement each other: retrieval decides what belongs in there, and the large window gives the model space for more extensive context packages.

RAG vs. Classic Full-Text and Semantic Search

Classic search provides a list of hits. RAG provides a formulated answer with source reference. Semantic search serves as a retriever component within many RAG systems, but it is not the entire system. Those who need a list for further manual review can use Enterprise Search; those who need a directly usable answer cannot.

What determines quality — Chunking, Embeddings, Hybrid Search, Reranking

The quality of a RAG pipeline rarely limits the language model alone—it sinks or swims with what the retriever delivers. Four adjustment screws account for the main share.

Chunk Size and Overlap

The typical working range is 128–512 tokens per chunk. For general use cases, recursive splitting set to 512 tokens with 10–20 % overlap has become the default. NVIDIA tested overlap values of 10 %, 15 %, and 20 % on FinanceBench and found 15 % to be optimal. The test ran on 1,024-token chunks; since overlap is specified as a percentage of the chunk size, the guideline is largely independent of chunk size and can therefore be applied to a 512-token basis—where 15 % corresponds to approximately 75 tokens. Below 10 % overlap, contextual information is lost at chunk boundaries. Above 25 %, noise arises from nearly identical matches. Fact-based questions benefit from smaller chunks, while conceptual summaries benefit from larger ones.

Choose an embedding model

Three selection criteria dominate: retrieval quality, latency, and hosting requirements. The MTEB benchmark serves as a guide—in the cloud segment, models from Cohere, OpenAI, or Google (Gemini Embedding); for self-hosted operation, open models like the BGE or Qwen embedding families; and smaller models from the E5 series for latency-critical applications. However, the MTEB top spots change rapidly, and MTEB v2 (from 2026) is not directly comparable to v1; specific rankings and scores become outdated within months. More important than the table position is this: MTEB scores do not reliably predict performance on one's own domain data. Before making a decision, it's worthwhile to conduct a benchmark on a small, representative sample of inquiries from your own knowledge base.

Hybrid Search and Reranking

Hybrid Search (Dense plus BM25) and a downstream reranking layer measurably improve the quality of search results beyond what a single method can achieve. The magnitude of this improvement can be seen in published benchmarks: An academic study on financial documents shows a Recall@5 of 0.816 and an MRR@3 of 0.605 for a two-stage pipeline consisting of hybrid retrieval and neural reranking; the same study finds that BM25 outperforms even state-of-the-art Dense Retrieval on financial documents. Individual real-world benchmarks (e.g., a published test of a FastAPI RAG system, Medium 2026) clearly confirm this trend—from around 60 % recall with pure Dense or BM25 to over 90 % in the full cascade with a reranker; such individual values are illustrative, not representative. In practice, this means that in almost every serious system, BM25 (via Elasticsearch or OpenSearch), a Dense retriever, and a reranker should be used together on the final 20–50 hits—not as standalone solutions.

Where RAG Fails in Practice

It happens quite often that RAG pipelines fail. The causes are almost never in the LLM, but in the retrieval path.

Silent Degradation: When the system responds, but incorrectly

A failed production RAG is recognized by the fact that nothing crashes. The pipeline continues to deliver answers, the dashboard shows healthy latency, and the evaluation results from the launch are still in the wiki.

Underneath, the system degrades on every relevant axis: new document versions are not indexed, the embedding space no longer fits the changed knowledge base, and irrelevant results appear more frequently.

Three anchors help with diagnosis: a firmly defined gold standard question set that runs periodically against production; samples of human evaluation instead of just automated metrics; tracking of faithfulness and context precision over time, not just at launch.

Typical error classes along the pipeline

During the indexing phase: unfavorable chunk boundaries that break tables, lists, or code; outdated documents; embeddings unsuited for the domain. During retrieval: Document-Level Retrieval Mismatch, where the system retrieves a chunk from a large document but not the relevant one; missing BM25 component for token-precise queries like product codes. During augmentation: too many chunks in the prompt, causing the model to lose context; missing source metadata, making citations impossible. During generation: the model responds with its own knowledge despite the context provided—a RAG hallucination that is only visible through faithfulness measurement.

How to measure if a RAG system is working

A common evaluation framework is RAGAS. The four most commonly used core dimensions each answer a different question.

Faithfulness checks if the generated answer is grounded in the retrieved documents and hasn't been hallucinated. This is the core metric for making RAG-specific hallucinations visible.

Answer Relevancy checks if the answer addresses the question asked—a faithful answer can miss the point.

Context Precision checks if the retrieved chunks are relevant to the question. Low values indicate retrieval problems.

Context Recall checks if all information needed for a complete answer has been retrieved. Low scores indicate gaps in the index or overly narrow retrieval.

Faithfulness and Context Precision target the retriever and the grounding of the answer to the sources. Answer Relevancy and Context Recall target completeness. RAGAS has since expanded its metric set (e.g., to include Noise Sensitivity and Response Groundedness); however, the four core metrics remain the pragmatic starting point. Tools like TruLens for traceability, DeepEval for unit-test-like checks, and Arize Phoenix for observability are used in addition. There is no formal ISO standard for RAG evaluation; task-specific benchmarks such as KILT, MS MARCO, FinanceBench, and Legalbench-RAG are established.

RAG in the B2B and Industrial Context — GDPR, On-Premise, and Concrete Use Cases

In the DACH (Germany, Austria, Switzerland) industrial environment, RAG is primarily used because it separates knowledge from the model—an architectural feature with immediate consequences for data protection and operations.

Typical application areas in industry and B2B

The Mittelstand-Digital Centre "Fokus Mensch" (Digital Centre SME "Focus on People") describes applications in industries such as mechanical engineering, electrical engineering, and logistics.

There, RAG links language models with internal and external data sources for real-time information. Lufthansa Industry Solutions cites internal knowledge management for process documentation, employee handbooks, and maintenance instructions, as well as B2B and B2C chatbots as typical fields.

Documented examples from production systems:

• Grainger (B2B MRO Distribution): RAG-based search system on Databricks Mosaic AI for 2.5 million products with approximately 400,000 daily product updates. Background: different buyer personas (e.g., electricians vs. mechanical engineers) expect industry-specific different results for a query like „terminals.“.
• Ramp uses RAG for industry classification by NAICS codes: customer information is vectorized, compared against a database of codes, and an LLM generates the final classification.
• Thomson Reuters uses RAG to provide support staff with relevant content from internal knowledge bases via a chat interface.
• The BVDW has taken up the pattern with its own white paper for the DACH market („Retrieval-Augmented Generation: Using knowledge specifically, generating answers precisely“).

GDPR, Data Residency, and On-Premises Operation

With RAG, confidential material remains outside the model weights: knowledge is retrieved from controlled repositories, not embedded into the model. This lowers the leak risk compared to fine-tuning, where training content can potentially be reconstructed from the weights. For the DACH industry, this leads to three architectural decisions that must be made early on: Where will the vector database be located (own infrastructure, EU cloud, US cloud)? Which generator will be used (cloud API, EU-hosted model, local open-weight model)? What source metadata will be stored, allowing for Audit and deletion requests are feasible? The separation of knowledge and model makes RAG an architecturally simpler solution for data residency requirements—but it does not replace clean data classification beforehand.

FAQ