A retrieval-augmented generation system is only as good as the passages it retrieves. If the right section of the statute or the right clause of the contract is not in the handful of passages the model reads, no amount of prompt engineering produces a correct, cited answer. In legal work that failure is not cosmetic. An answer that cites the wrong section, or the wrong version of the right section, is worse than no answer.
This article uses two worked examples from English housing law and one from a commercial contract to show why legal documents defeat single-method retrieval, how hybrid search fixes it, and why reciprocal rank fusion is the fusion method we reach for first. The examples are illustrations built on public law, not client data. Where the law is cited, the sources are linked so you can check them.
Why does legal RAG need hybrid search? Two queries, two failure modes
Consider a knowledge base containing the Housing Act 1988 as amended, the Renters' Rights Act 2025, and a firm's own guidance notes. A lawyer asks two questions in the same afternoon.
Query A: "Housing Act 1988 section 21 notice requirements"
This is an identifier query. The lawyer knows exactly which provision they want. Keyword retrieval, typically BM25, excels here: the tokens Housing Act 1988, section 21 and notice appear in the target passage and almost nowhere else with that density. Embedding-based retrieval tends to struggle. Embeddings compress meaning, and the meaning of section 21 is close to the meaning of section 8, section 21A and section 21B, all of which concern notices and possession. The exact number is a small feature in a large vector, so the wrong neighbouring section can outrank the right one.
Query B: "Can a landlord end a tenancy without giving a reason?"
This is a conceptual query. None of its words appear in the statute in that form. The Act speaks of recovering possession of a dwelling-house let on an assured shorthold tenancy, not of ending a tenancy without a reason. Keyword retrieval has nothing to match and returns passages about reasons, tenancies or landlords in general. Embedding retrieval handles it well, because the plain-English question and the statutory language sit close together in meaning space even though they share no vocabulary.
Query C: "What is the liability cap in clause 14.3?"
Contracts add a third pattern: internal references and defined terms. Clause numbers, schedule references and capitalised defined terms such as Permitted Use or Services are identifiers with no independent meaning. Embeddings treat clause 14.3 as roughly interchangeable with clause 14.2. Keyword retrieval treats them as different strings, which is what the lawyer needs.
The pattern is consistent. Legal text is dense with identifiers that carry no meaning of their own, and legal questions are often phrased in language the source never uses. A retrieval system that is good at only one of these will fail lawyers on a large share of their real queries.
How does hybrid search work, and why is merging the results hard?
The obvious fix is hybrid search: run both retrievers and combine the results. The difficulty is in the combining. BM25 returns scores on an unbounded scale that depends on term frequencies and document length. A vector store returns cosine similarities between roughly 0 and 1, or distances on some other scale entirely. Adding the two together is meaningless, and normalising them requires assumptions that break as the corpus changes. A firm that adds ten thousand pages of case law to its knowledge base should not have to re-tune a weighting formula.
Reciprocal rank fusion sidesteps the problem by ignoring scores altogether. It uses only the rank of each passage in each list.
What is reciprocal rank fusion, and why does RRF work for legal search?
For each passage d that appears in any of the result lists, the fused score is the sum over the lists of one divided by a constant plus the passage's rank in that list:
RRF(d) = ∑i 1 / (k + ranki(d))The constant k is conventionally 60, the value used in the original 2009 paper by Cormack, Clarke and Buettcher. It dampens the advantage of a first-place rank so that a passage ranked highly by both retrievers beats a passage ranked first by one and ignored by the other. Passages that appear in only one list still receive a score and can still surface, which matters for identifier queries where the semantic list may be empty of the right answer.
A worked RRF example with k = 60
Suppose four passages appear in the top results for Query A, with the following ranks in each list. A dash means the passage was not returned by that retriever at all.
| Passage | BM25 rank | Embedding rank | RRF score, k = 60 |
|---|---|---|---|
| W: section 21 as amended, notice requirements | 3 | 2 | 1/63 + 1/62 = 0.0320 |
| X: section 21, original 1988 text | 1 | 5 | 1/61 + 1/65 = 0.0318 |
| Z: section 8, grounds for possession | — | 1 | 1/61 = 0.0164 |
| Y: firm guidance note on serving notices | 2 | — | 1/62 = 0.0161 |
Two things happen that a single retriever could not do. Passage W, which neither retriever ranked first, wins because both retrievers agreed it was relevant. Passage Z, which the embedding retriever placed first, drops to third because the keyword retriever found no support for it, and section 8 is indeed the wrong section for this query. The fusion rewards consensus and punishes confident isolated errors, which is exactly the behaviour you want when one retriever is prone to confusing neighbouring sections.
Note also that the fused scores are small and close together. That is normal. RRF scores are for ordering, not for judging confidence, and they should never be shown to a user as a relevance percentage.
What does RRF not solve in legal RAG?
Fusion solves the merging problem. It does not solve four other problems that legal retrieval raises, and an honest system design addresses each of them separately.
Reranking
RRF gives a good top twenty or top fifty. It does not read the query and the passage together. A cross-encoder reranker does, and applying one to the fused shortlist typically produces a sharper top five than either retriever or the fusion alone. The cost is latency, so the shortlist should be small enough to rerank in well under a second.
Chunking at legal boundaries
If passages are cut every five hundred tokens regardless of structure, a section can be split mid-sentence and a clause can lose its heading. Legal documents have natural units: sections and subsections, clauses and sub-clauses, schedules, paragraphs of a judgment. Chunk at those boundaries and attach the heading path as metadata, so the passage for Query C carries Clause 14: Liability, 14.3: Cap rather than an orphaned paragraph beginning "shall not exceed".
Temporal validity
This is the problem that makes legal retrieval genuinely different. Query B has a different correct answer depending on the date and the tenancy. Under the Housing Act 1988 a landlord in England could serve a section 21 notice and recover possession without stating a reason. The Renters' Rights Act 2025 abolished section 21 notices in England from 1 May 2026, with transitional rules for notices already served, as set out by Shelter and summarised for landlords by the National Residential Landlords Association. A retrieval system that fuses ranks perfectly but returns the 2019 version of the section alongside the 2026 position, with no indication of which applies, will produce a fluent and wrong answer.
The fix is metadata, not fusion. Every passage needs the version of the instrument it comes from, the dates that version was in force, and the jurisdiction it applies to. Retrieval then filters on the relevant date and jurisdiction before ranking, and the answer cites the version it used. Passages W and X in the worked example are the same section in two versions, and the system should know which one the lawyer's matter falls under.
Permissions and privilege
A firm's knowledge base mixes public law with privileged matter files. Retrieval must carry the user's permissions into the query so that a passage the user may not open cannot appear in a shortlist, let alone in an answer. This is a property of the index and the query filter, and it must hold before fusion, not after.
How do you evaluate retrieval for legal RAG?
Retrieval should be evaluated on its own, before anyone looks at generated answers. A fluent answer built on the wrong passage hides the retrieval failure that caused it. The method we use is straightforward and any firm can run it.
- Build a graded question set. Collect fifty to two hundred real questions from the lawyers who will use the system, covering identifier queries, conceptual queries and date-sensitive queries. For each, a lawyer marks the passages that constitute a correct source. This is the expensive step and the only one that matters.
- Measure retrieval alone. For keyword only, embedding only and the fused list, record how often a correct passage appears in the top five and top ten, and the rank of the first correct passage. Break the results down by query type. Expect keyword to win on identifiers, embeddings to win on concepts, and fusion to be close to the best of both on each.
- Test the date filter separately. Ask the same conceptual question with different matter dates and check that the version returned changes accordingly.
- Then evaluate answers. Only once retrieval is sound, check that generated answers stay within the retrieved evidence, cite the passage and version they used, and decline when the sources do not support an answer.
Run the same set again whenever the corpus, the embedding model or the chunking changes. Retrieval quality drifts silently, and the graded set is the only instrument that detects it.
What does this mean for a law firm considering RAG?
The retrieval layer is where a legal RAG project succeeds or fails, and hybrid search with reciprocal rank fusion is the sensible default for it. It handles the two kinds of question lawyers actually ask, it needs no score tuning as the knowledge base grows, and its behaviour is easy to explain to the people who have to trust it. It is not the whole design. Reranking, structure-aware chunking, version and jurisdiction metadata, and permission-aware indexing each remove a distinct failure mode, and the graded question set is what proves the system works before anyone relies on it.
If you are scoping a document system for legal or compliance work, our legal and compliance document AI page describes the workflow, and how it works sets out the steps from a first conversation to a tested pilot. Bring a few real questions and the documents they should be answered from. That is where every project starts.
Related reading
Retrieval is only half of a document system; the other half is trusting what was extracted in the first place. Invoice OCR: validation before automation walks one scanned invoice through the checks that decide whether a field can be trusted, with a worked example of a low-confidence reference. All articles are listed on the Insights page.