If you're building a RAG system and getting wrong answers despite using state-of-the-art embeddings, your problem isn't the LLM. It's retrieval. Vector-only systems fail predictably on exact matches like product codes, order IDs, precise terminology, while keyword systems miss paraphrases and synonyms. Hybrid search combines vector embeddings with BM25 keyword matching, then uses Reciprocal Rank Fusion to merge results. Add a reranker as a second-stage filter, and you catch what pure vector search silently drops. This guide shows you exactly how to build and measure that pipeline.
Why Does My RAG System Give Wrong Answers?
Most developers blame the language model when their RAG system hallucinates or returns irrelevant answers. That's almost always the wrong diagnosis. The LLM can only work with what you feed it. If your retrieval step doesn't surface the right chunks, the best model in the world will fabricate an answer or work from irrelevant context.
Retrieval is the bottleneck. In production RAG systems, failed queries typically break down this way: retrieval problems account for the majority of errors, with the model itself responsible for a smaller fraction. You can't fix a retrieval problem by switching from GPT-4 to Claude or tweaking your prompt.
Vector-only retrieval has predictable blind spots. It excels at semantic similarity but chokes on exact identifiers. If a user asks "What's the status of order #4782B?" and your system only uses embeddings, it might return chunks about order processing in general, missing the specific order entirely. The embedding space treats "order 4782B" and "order 4783B" as nearly identical vectors.
Vector Search vs Keyword Search for RAG Systems
Vector search and keyword search fail in opposite scenarios. Understanding this mirror-image weakness is critical to fixing your RAG system.
Vector embeddings map text into high-dimensional space where semantically similar content clusters together. When someone asks "How do I reset my password?" your vector search will correctly surface chunks about "account recovery" and "credential reset" even if those exact words don't appear. This semantic matching is powerful for natural language queries and paraphrases.
But vector search collapses on exact matches. Product codes, serial numbers, API endpoints, specific error messages, technical identifiers. They don't embed meaningfully. The model hasn't learned that "SKU-9847" is fundamentally different from "SKU-9848" in a way that matters to your users. In benchmark tests, vector-only retrieval on datasets with alphanumeric identifiers shows recall rates dropping to 40-55% compared to 85-90% on purely semantic queries.
BM25 keyword search is the inverse. It's a probabilistic ranking function that scores documents based on term frequency and inverse document frequency. If your query contains "order #4782B," BM25 will nail that exact match every time. It doesn't care about semantics, just token overlap.
The failure mode: BM25 misses paraphrases completely. A query for "password reset" won't match a document that only says "recover your account credentials." Zero shared tokens means zero relevance score, even if the content is exactly what the user needs.
How Hybrid Search and Reranking Work Together
Hybrid search runs vector and keyword retrieval in parallel, then fuses the results. You're covering both failure modes simultaneously. When a query contains exact identifiers, BM25 catches them. When it's purely semantic, vector search handles it. For mixed queries, both contribute.
The fusion step typically uses Reciprocal Rank Fusion (RRF). Instead of trying to normalize scores from two completely different algorithms (which is mathematically sketchy), RRF works with ranks. Each retriever returns its top-k results ranked by relevance. RRF assigns a score to each document based on its rank position across all retrievers.
The formula is simple: for each document, sum 1/(k + rank) across all retrievers where it appears. The constant k (usually 60) prevents top-ranked documents from dominating too heavily. Documents that appear in both result sets get boosted naturally. This approach is surprisingly effective and doesn't require tuning weights between retrievers.
Reranking is your second-stage filter. After RRF gives you a merged candidate set (typically 20-50 documents), you pass the query and each candidate through a cross-encoder model. Unlike bi-encoders that embed query and document separately, cross-encoders process them jointly and output a relevance score. This is computationally expensive but much more accurate.
Cross-encoders like cross-encoder/ms-marco-MiniLM-L-6-v2 are trained specifically for relevance ranking. They catch nuances that embedding similarity misses. In controlled tests on the MS MARCO dataset, reranking the top 20 candidates improves MRR@10 (Mean Reciprocal Rank at 10) by approximately 18-22 percentage points compared to using retrieval scores alone.
How to Implement Hybrid Search with Reranking in RAG
Here's a concrete implementation using Python with common libraries. This example uses FAISS for vector search, Rank-BM25 for keyword search, and Sentence-Transformers for reranking.
Step 1: Set Up Your Retrievers
Install dependencies first:
pip install faiss-cpu sentence-transformers rank-bm25 numpyInitialize both retrievers with your document corpus:
from sentence_transformers import SentenceTransformer, CrossEncoder
import faiss
import numpy as np
from rank_bm25 import BM25Okapi
import re
# Your document corpus
documents = [
"Order #4782B was shipped on March 15th",
"To reset your password, visit the account recovery page",
"Product SKU-9847 is currently out of stock",
# ... more documents
]
# Vector retriever setup
embedding_model = SentenceTransformer('all-MiniLM-L6-v2')
doc_embeddings = embedding_model.encode(documents)
dimension = doc_embeddings.shape[1]
index = faiss.IndexFlatIP(dimension) # Inner product for cosine similarity
faiss.normalize_L2(doc_embeddings)
index.add(doc_embeddings)
# BM25 retriever setup
def tokenize(text):
return re.findall(r'\w+', text.lower())
tokenized_docs = [tokenize(doc) for doc in documents]
bm25 = BM25Okapi(tokenized_docs)Step 2: Implement Reciprocal Rank Fusion
Create functions to retrieve from both systems and fuse results:
def vector_search(query, k=20):
query_embedding = embedding_model.encode([query])
faiss.normalize_L2(query_embedding)
scores, indices = index.search(query_embedding, k)
return [(idx, score) for idx, score in zip(indices[0], scores[0])]
def bm25_search(query, k=20):
tokenized_query = tokenize(query)
scores = bm25.get_scores(tokenized_query)
top_indices = np.argsort(scores)[::-1][:k]
return [(idx, scores[idx]) for idx in top_indices]
def reciprocal_rank_fusion(results_list, k=60):
"""
results_list: list of [(doc_id, score), ...] from different retrievers
k: constant for RRF formula (typically 60)
"""
rrf_scores = {}
for results in results_list:
for rank, (doc_id, _) in enumerate(results):
if doc_id not in rrf_scores:
rrf_scores[doc_id] = 0
rrf_scores[doc_id] += 1 / (k + rank + 1)
sorted_docs = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
return [doc_id for doc_id, score in sorted_docs]Step 3: Add Reranking
Use a cross-encoder to rerank the fused results:
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
def hybrid_search_with_reranking(query, top_k=5):
# Get candidates from both retrievers
vector_results = vector_search(query, k=20)
bm25_results = bm25_search(query, k=20)
# Fuse with RRF
fused_doc_ids = reciprocal_rank_fusion([vector_results, bm25_results])
# Take top 20 for reranking (balance quality vs latency)
candidate_ids = fused_doc_ids[:20]
candidate_docs = [documents[i] for i in candidate_ids]
# Rerank with cross-encoder
pairs = [[query, doc] for doc in candidate_docs]
rerank_scores = reranker.predict(pairs)
# Sort by reranker scores
ranked_indices = np.argsort(rerank_scores)[::-1]
final_results = [(candidate_ids[i], candidate_docs[i], rerank_scores[i])
for i in ranked_indices[:top_k]]
return final_resultsStep 4: Use It in Your RAG Pipeline
Now integrate this into your RAG system:
query = "What's the status of order #4782B?"
results = hybrid_search_with_reranking(query, top_k=3)
# Feed top results to your LLM
context = "\n\n".join([doc for _, doc, _ in results])
prompt = f"Answer the question based on this context:\n\n{context}\n\nQuestion: {query}"
# Send to your LLM (OpenAI, Anthropic, etc.)
# answer = llm.generate(prompt)For production systems, you'll want to use LangChain or LlamaIndex which provide built-in hybrid retrieval components. But understanding this implementation helps you debug when things go wrong. If you're building more complex systems with multiple agents, the patterns in running multiple AI agents at scale become relevant.
How to Measure RAG Retrieval Performance with Recall and MRR
You can't improve what you don't measure. Most developers ship RAG systems without proper evaluation, then wonder why users complain. You need metrics that quantify retrieval quality before the LLM ever sees the context.
Start by building a test set. Take 50-100 real user queries and manually label which documents should be retrieved for each. Yes, this is tedious. Do it anyway. Without ground truth labels, you're flying blind.
Recall@k measures what percentage of relevant documents appear in your top-k results. If a query has 3 relevant documents in your corpus and 2 of them appear in your top-10 results, recall@10 is 0.67. This tells you if your system is even surfacing the right content.
Mean Reciprocal Rank (MRR) measures how high the first relevant result appears. If the first relevant document is rank 1, you score 1.0. If it's rank 2, you score 0.5. If it's rank 3, you score 0.33. Average this across all queries. MRR tells you if users will see good results immediately or have to dig through irrelevant content.
Normalized Discounted Cumulative Gain (nDCG) is more sophisticated. It accounts for both position and degree of relevance (if you have graded labels like "highly relevant" vs "somewhat relevant"). Documents at higher ranks contribute more to the score, with a logarithmic discount. An nDCG@10 of 0.85 means your system is doing quite well.
Here's evaluation code you can actually run:
def recall_at_k(retrieved_ids, relevant_ids, k):
retrieved_k = set(retrieved_ids[:k])
relevant = set(relevant_ids)
if len(relevant) == 0:
return 0.0
return len(retrieved_k & relevant) / len(relevant)
def mean_reciprocal_rank(retrieved_ids, relevant_ids):
for rank, doc_id in enumerate(retrieved_ids, start=1):
if doc_id in relevant_ids:
return 1.0 / rank
return 0.0
def evaluate_retrieval(test_queries, retrieval_function):
recall_5_scores = []
recall_10_scores = []
mrr_scores = []
for query, relevant_ids in test_queries:
results = retrieval_function(query)
retrieved_ids = [doc_id for doc_id, _, _ in results]
recall_5_scores.append(recall_at_k(retrieved_ids, relevant_ids, 5))
recall_10_scores.append(recall_at_k(retrieved_ids, relevant_ids, 10))
mrr_scores.append(mean_reciprocal_rank(retrieved_ids, relevant_ids))
print(f"Recall@5: {np.mean(recall_5_scores):.3f}")
print(f"Recall@10: {np.mean(recall_10_scores):.3f}")
print(f"MRR: {np.mean(mrr_scores):.3f}")
# Example usage
test_queries = [
("order #4782B status", [0, 5, 12]), # doc IDs that should be retrieved
("how to reset password", [1, 8]),
# ... more test cases
]
evaluate_retrieval(test_queries, hybrid_search_with_reranking)Run this on three configurations: vector-only, BM25-only, hybrid with reranking. You'll see concrete numbers showing which approach works best for your specific use case. In knowledge base applications with mixed query types, hybrid systems typically achieve recall@10 scores in the 0.75-0.85 range, compared to 0.55-0.65 for vector-only approaches.
Understanding different RAG architectures helps you choose the right evaluation strategy for your system's complexity level.
Common Implementation Mistakes to Avoid
The biggest mistake is shipping vector-only systems without testing on edge cases. Developers fall in love with embeddings because they're conceptually elegant, then discover in production that users searching for specific codes or IDs get garbage results. Always test with queries containing exact identifiers, version numbers, alphanumeric codes before launch.
Skipping evaluation is the second killer. You need those metrics before you deploy. Anecdotal testing ("I tried a few queries and they looked good") doesn't cut it. Build that test set of 50-100 labeled queries and run the evaluation code above. If your recall@10 is below 0.70, you have a retrieval problem that will tank user satisfaction.
Reranking too many candidates wastes money and adds latency. Cross-encoders are expensive because they process each query-document pair individually. Reranking 100 documents might add 300-500ms of latency depending on your model and hardware. Reranking 20 documents adds roughly 80-120ms. Find the sweet spot where you're catching quality improvements without destroying response times.
Look, not tuning your chunk size is another common error. If your chunks are too large (over 1000 tokens), you're feeding the LLM irrelevant context that dilutes the signal. Too small (under 200 tokens), and you're fragment
Get a free AI-powered SEO audit of your site
We'll crawl your site, benchmark your local pack, and hand you a prioritized fix list in minutes. No call required.
Run my free audit