Skip to content
Content

Unleashing AI's Full Potential: How Contextual Retrieval Transforms Knowledge Bases

March 3, 2025
standard rag

Key Points - Research suggests Contextual Retrieval improves RAG by adding context to chunks,

reducing retrieval failures by up to 49%. - It seems likely that combining Contextual Embeddings and BM25, plus reranking, can enhance accuracy by 67%. - The evidence leans toward using Claude with prompt caching for efficient implementation, especially for larger knowledge bases.

What is Contextual Retrieval and How Does It Help AI?

Understanding RAG and Its Challenges
Retrieval Augmented Generation (RAG) is a technique used to enhance AI models, like chatbots, by pulling relevant information from a knowledge base and adding it to user queries. This is crucial for applications like customer support or legal analysis, where specific knowledge is key. However, traditional RAG often loses context when breaking down large knowledge bases into smaller chunks, making it hard to retrieve the right information. For example, a chunk might say "revenue grew by 3%" without specifying the company or time, leading to confusion. How Contextual Retrieval Solves This
Contextual Retrieval is a new method that improves RAG by adding context back to each chunk. It uses two techniques:
  • Contextual Embeddings: Adds AI-generated context to chunks before turning them into numerical representations (embeddings) for retrieval.
  • Contextual BM25: Enhances exact term matching by including this context in the indexing process.
This approach reduces retrieval failures by up to 49%, and when combined with reranking—where the most relevant chunks are further filtered—it can improve accuracy by 67%. An unexpected detail is that for smaller knowledge bases (under 200,000 tokens, about 500 pages), you can simply include the entire knowledge base in the prompt, using prompt caching for efficiency, which cuts costs by up to 90%. Getting Started with Implementation
You can implement Contextual Retrieval using Claude, an AI model, with prompt caching to make it cost-effective. The process involves generating context for each chunk and then embedding it for retrieval. For detailed steps, check out the cookbook guide provided.

Detailed Analysis of Contextual Retrieval and RAG Improvements

This section provides a comprehensive exploration of Contextual Retrieval, expanding on the direct answer with detailed insights into its mechanisms, performance, and implementation considerations. The analysis is grounded in the provided article and aims to offer a professional, thorough understanding for developers and AI practitioners.

Introduction to RAG and the Context Challenge

Retrieval Augmented Generation (RAG) is a pivotal technique in AI, particularly for applications requiring domain-specific knowledge, such as customer support chatbots needing business details or legal analyst bots accessing past case law. RAG operates by preprocessing a knowledge base—breaking it into chunks (typically a few hundred tokens), converting these into vector embeddings using an embedding model, and storing them in a vector database for semantic similarity searches. At runtime, a user query triggers a search for the most relevant chunks, which are then appended to the prompt for the generative model.
diagram
However, traditional RAG faces a significant challenge: context loss. When documents are split into chunks, individual pieces may lack sufficient context, leading to retrieval failures. For instance, consider a financial knowledge base with a chunk stating, "The company's revenue grew by 3% over the previous quarter." Without additional context, such as the company name (e.g., ACME Corp) or the time period (e.g., Q2 2023), retrieval systems may struggle to match this to relevant queries, resulting in inaccurate or incomplete responses.

Contextual Retrieval: A Solution to Context Loss

Contextual Retrieval addresses this issue by enhancing the retrieval step with two sub-techniques: Contextual Embeddings and Contextual BM25. This method involves prepending chunk-specific explanatory context to each chunk before embedding and indexing, significantly improving retrieval accuracy.
diagram
Mechanism of Contextual Retrieval
  1. Chunk Contextualization:
    The process begins with generating context for each chunk using an AI model, specifically Claude in this case. A prompt is designed to instruct Claude to provide a concise, chunk-specific context based on the entire document. For example, given a document and a chunk like "The company's revenue grew by 3% over the previous quarter," Claude might generate: "This chunk is from an SEC filing on ACME Corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter." This contextualized chunk is then used for further processing.     The prompt used is structured as follows: 
    <document> {{WHOLE_DOCUMENT}} </document>
Here is the chunk we want to situate within the whole document
<chunk> {{CHUNK_CONTENT}} </chunk>
Please give a short succinct context to situate this chunk
within the overall document for the purposes of improving search retrieval of the chunk.
Answer only with the succinct context and nothing else.
    The resulting context, typically 50-100 tokens, is prepended to the chunk.
  2. Contextual Embeddings:
    The contextualized chunks are converted into vector embeddings using an embedding model. This step ensures that the embeddings capture not only the chunk's content but also its contextual relevance, improving semantic similarity searches.
  3. Contextual BM25:
    BM25, a ranking function based on TF-IDF (Term Frequency-Inverse Document Frequency), is used for lexical matching, particularly effective for queries with unique identifiers or technical terms. By applying BM25 to contextualized chunks, it enhances exact match retrieval, complementing the semantic capabilities of embeddings.
  4. Retrieval Process:
    During retrieval, both embedding similarity and BM25 scores are leveraged to find the top relevant chunks. The results are combined and deduplicated using rank fusion techniques, ensuring comprehensive coverage.
  5. Reranking for Further Optimization:
    Reranking is an additional step where, after initial retrieval (e.g., top 150 chunks), a reranking model scores each chunk based on relevance to the query and selects the top-K (e.g., top 20) for the final prompt. This reduces latency and cost by processing fewer, more relevant chunks. The article tested with the Cohere reranker, noting that other options like Voyage's reranker exist, though not evaluated in this study.
diagram

Performance Improvements and Experimental Insights

The article reports significant performance gains from Contextual Retrieval across various knowledge domains, including codebases, fiction, ArXiv papers, and science papers. The evaluation metric used was 1 minus recall@20, measuring the percentage of relevant documents not retrieved within the top 20 chunks. Key findings include:
  • Contextual Embeddings reduced the top-20-chunk retrieval failure rate by 35% (from 5.7% to 3.7%).
  • Combining Contextual Embeddings and Contextual BM25 reduced it by 49% (to 2.9%).
  • With reranking, the failure rate dropped to 1.9%, a 67% reduction from the baseline.
These improvements were consistent across embedding models, with Gemini Text 004 and Voyage embeddings performing particularly well. The study also experimented with different numbers of retrieved chunks (5, 10, 20), finding that 20 chunks yielded the best performance, though more chunks could increase distraction for the model, suggesting a trade-off.

Implementation Considerations

Implementing Contextual Retrieval requires attention to several factors:
  • Chunk Boundaries: The choice of chunk size, boundary, and overlap affects retrieval performance. Optimal settings may vary by use case.
  • Embedding Model: While Contextual Retrieval improves performance across models, Gemini and Voyage embeddings showed superior results.
  • Custom Contextualizer Prompts: Tailoring prompts to specific domains (e.g., including glossaries for technical terms) can enhance effectiveness.
  • Number of Chunks: Passing more chunks (e.g., 20) increases relevance but may impact model performance due to distraction. Experimentation is recommended.
  • Evaluation: Always run evaluations to assess response generation, distinguishing between context and chunk content.

Cost and Efficiency with Prompt Caching

Contextual Retrieval is uniquely cost-effective with Claude, thanks to prompt caching. This feature allows caching frequently used prompts between API calls, reducing latency by over 2x and costs by up to 90%. For example, assuming 800-token chunks, 8k-token documents, 50-token context instructions, and 100 tokens of context per chunk, the one-time cost to generate contextualized chunks is $1.02 per million document tokens. This efficiency is particularly beneficial for large knowledge bases, where traditional RAG without context would be less effective. For smaller knowledge bases (under 200,000 tokens, about 500 pages), the article notes an alternative: including the entire knowledge base in the prompt, leveraging prompt caching for speed and cost savings. This approach eliminates the need for RAG, but as knowledge bases grow, Contextual Retrieval becomes essential.

Practical Application and Getting Started

Developers can implement Contextual Retrieval using the provided cookbook guide, which includes step-by-step instructions and code snippets. This resource is particularly useful for experimenting with the technique across different domains and use cases, ensuring adaptability to specific needs.

Conclusion

Contextual Retrieval represents a significant advancement in RAG systems, addressing the critical issue of context loss and enhancing retrieval accuracy. By combining Contextual Embeddings, Contextual BM25, and optional reranking, it achieves up to 67% improvement in retrieval performance, making it a vital tool for AI applications with large knowledge bases. Its cost-effectiveness, enabled by Claude and prompt caching, further underscores its practicality, offering a scalable solution for modern AI challenges.

Key Citations

Cheers!