Local knowledge is a deployment choice, not a quality badge.
Organise collections, add context, combine BM25 and vector search, rerank a small result set and evaluate the pipeline before calling it useful.
Short answer
A practical local pipeline starts with named collections and short descriptions of what belongs in each one. It indexes exact language for lexical search, embeddings for semantic similarity and enough metadata to enforce product and access boundaries. At query time, generate candidates from both views, fuse them, remove duplicates and rerank a bounded set before loading full documents. QMD is a useful field implementation of this pattern for Markdown collections. Running locally is one deployment option. It does not make retrieval accurate, current, authorised or secure by itself.
Key points
- 01
Collections and context labels give the search system a map before it ranks passages.
- 02
Lexical search, vector search and reranking solve different retrieval failures.
- 03
Local execution changes where data and models run; quality still needs representative evaluations.
01 / SITUATION
A folder of documents is not yet a knowledge system
An industrial team may have procedures, maintenance notes and supplier manuals in Markdown or exported text. A software company may have product docs, release notes and internal decisions. Putting those files behind a search command creates access, but it does not establish which source is authoritative, who may see it, how fresh it is or which collection answers the question.
The useful pipeline begins before embeddings. It defines collections, ownership, update rules and contextual labels. Retrieval can then combine exact product language with semantic similarity without asking the model to guess what every folder means.
02 / MECHANISM
From collections to evaluated evidence
Keep each stage observable so a weak answer can be traced to the source, index, query or ranking step.
- 01Define collections
Group documents by a stable business boundary such as product, site, team or source authority. Record an owner and an update path for each collection.
- 02Add collection context
Describe what the collection contains, who it serves and when it should or should not be searched. QMD contexts provide one concrete way to attach this map to a document tree.
- 03Build complementary indexes
Keep a lexical index for codes, labels and exact phrases, plus a vector index for paraphrases and conceptual similarity. Version the inputs and embedding configuration.
- 04Retrieve within the boundary
Apply collection, role, locale and product filters first. Generate lexical and semantic candidates without loading unrelated documents into model context.
- 05Fuse, deduplicate and rerank
Merge the candidate lists with a stable rule, collapse repeated content and rerank a bounded set against the full question and current product context.
- 06Inspect, evaluate and refresh
Test representative queries, negatives and outdated sources. Record which stage failed, rebuild changed indexes and keep a clear fallback when evidence is missing.
03 / In real work
Industrial procedures
Separate approved procedures from supplier manuals and maintenance notes. An exact equipment code finds the right family; semantic search helps when the operator describes the symptom instead.
Accounting knowledge
Keep firm-wide policy apart from client-specific material. Access filters select the permitted collection before either retriever runs.
Product documentation
Index the current guide, release notes and known limitations as distinct sources. A deployment job refreshes only the collections affected by a documentation change.
Common failure modes
- Pointing the indexer at every folder without defining ownership or authority.
- Treating local execution as proof of accuracy, confidentiality or access control.
- Applying document permissions after candidate text has already entered the prompt.
- Refreshing text files without rebuilding stale lexical or vector indexes.
- Judging the pipeline from a few successful demos instead of a representative query set.
Implementation checklist
- Every collection has a purpose, owner and update rule.
- Context labels explain inclusion and exclusion boundaries.
- Access and product filters run before retrieval.
- Lexical and semantic indexes can be inspected separately.
- Fusion, deduplication and reranking are deterministic or versioned.
- Evaluations include exact terms, paraphrases, negatives and stale documents.
- Missing evidence produces an honest fallback instead of a generated guess.
What this approach does not claim
- QMD is an engineering reference, not evidence that one configuration fits every corpus.
- Local models and hardware can change latency, coverage and operating cost without improving relevance.
- A search pipeline cannot repair contradictory documents or unclear source ownership.
Sources and reference types
- 01The Probabilistic Relevance Framework: BM25 and BeyondRobertson and Zaragoza · 2009Research
- 02ColBERTv2: Effective and Efficient Retrieval via Lightweight Late InteractionSanthanam et al. · 2022Research
- 03RAGAS: Automated Evaluation of Retrieval Augmented GenerationEs et al. · 2024Research
- 04QMD: query Markdown with hybrid searchTobi Lütke · 2026Field practice
Practical questions
Does local retrieval mean the whole assistant stays local?
No. Indexing and search may run locally while generation or tools run elsewhere. Map every data path and deployment boundary explicitly.
Should we start with vectors?
Start with a lexical baseline and representative questions. Add semantic retrieval when it fixes paraphrases or conceptual misses that the baseline exposes.
When should a collection be reindexed?
When source content, parsing, chunking or embedding configuration changes. The refresh policy should match how quickly each source becomes stale.