Hybrid retrieval: search broadly, rerank precisely.
A practical retrieval pipeline using permission filters, BM25, vectors, fusion, deduplication, reranking and separate evaluation.
Short answer
Use permission and product-scope filters before retrieval. Run lexical search for exact terms and semantic search for meaning, merge their candidates, remove duplicates and rerank a small set against the actual question. Send only the best evidence to the model. Evaluate retrieval, relevance, faithfulness and final task outcome separately. Hybrid retrieval is not automatically better: it earns its complexity only when representative queries show that each stage fixes a real failure mode.
Key points
- 01
Apply access filters before candidate generation.
- 02
Use lexical and semantic search for complementary failures, not fashion.
- 03
Evaluate every stage separately so answer quality does not hide retrieval defects.
01 / SITUATION
Product language contains both exact identifiers and fuzzy intent
Users quote error codes, field labels and acronyms that lexical search handles well. They also describe goals in language that does not appear in the documentation, where semantic retrieval can help. A single retriever often misses one of these forms.
The solution is not to concatenate every result. Fusion and reranking should produce a small, diverse evidence set. Otherwise the generator must resolve duplicate, weak or contradictory chunks at higher cost.
02 / MECHANISM
A production retrieval pipeline
Each stage should have an observable input, output and reason to exist.
- 01Filter
Apply tenant, product version, role, locale and source-status filters before search. Security is not a reranking feature.
- 02Generate lexical candidates
Use BM25 or an equivalent for exact terms, identifiers, rare vocabulary and clear keyword matches.
- 03Generate semantic candidates
Use embeddings for paraphrases and conceptual similarity. Keep model, chunking and version information reproducible.
- 04Fuse and deduplicate
Merge ranks with a stable method, collapse repeated chunks and retain source diversity when it helps the task.
- 05Rerank
Score a bounded candidate set against the full question and product context. Avoid an expensive reranker when the first stage is already decisive.
- 06Evaluate and cache
Measure retrieval recall and precision, then answer faithfulness and task completion. Cache stable stages with explicit invalidation.
03 / In real work
Exact code, broad explanation
BM25 finds the error identifier while semantic search finds the procedure described with different words. Reranking keeps the relevant pair.
Role-specific guide
Metadata filtering removes guides for other roles before either retriever runs.
Repeated documentation
Deduplication prevents the same release note copied across pages from dominating the context.
Common failure modes
- Applying access filters after candidate text has entered model context.
- Using only vector search for exact codes and product labels.
- Increasing top-k until the correct chunk appears, then flooding generation.
- Evaluating only final answers and missing a fragile retrieval stage.
- Treating an LLM-based evaluator as ground truth without calibrated examples.
Implementation checklist
- A representative query set includes exact terms, paraphrases and negatives.
- Metadata filters are deterministic and tested.
- Lexical and semantic candidate sets are logged in redacted form.
- Fusion and deduplication rules are stable.
- Reranking operates on a bounded set.
- Retrieval and generation metrics are reviewed separately.
- Source changes and caches have an invalidation policy.
What this approach does not claim
- Hybrid search adds infrastructure, tuning and evaluation work.
- Public benchmark gains may not transfer to a specialised product corpus.
- Rerankers can still prefer fluent but wrong context and need domain evaluation.
Sources and reference types
- 01The Probabilistic Relevance Framework: BM25 and BeyondRobertson and Zaragoza · 2009Research
- 02ColBERTv2: Effective and Efficient Retrieval via Lightweight Late InteractionSanthanam et al. · 2022Research
- 03RAGAS: Automated Evaluation of Retrieval Augmented GenerationEs et al. · 2024Research
- 04QMD: query Markdown with hybrid searchTobi Lütke · 2026Field practice
Practical questions
Do we always need vectors?
No. Start with a baseline. A small, well-labelled corpus with exact product terms may be served well by lexical search.
How large should the final context be?
As small as possible while preserving the evidence needed for the answer. Determine it from evaluations, not a universal token count.
Where does QMD fit?
QMD is a useful field implementation of hybrid Markdown search. It is an engineering reference, not scientific proof that one pipeline fits every corpus.