RAG query planning: why a story is not a search query

A lawyer, a paralegal or a member of the public rarely types only "deposit protection requirements". They describe when they moved in, what they paid, what the landlord did and what happened next. The actual question comes at the end. Sometimes there are two or three questions.

Sending that whole story to a retriever as one query can produce passages about the right topic without covering the issues the person needs answered. A more capable retriever helps, but it does not explicitly separate those issues.

This article explains RAG query planning, also called query decomposition when the task is split into sub-questions. A model first identifies the evidence needs inside the message. Retrieval runs for each need, and a coverage check records what the sources do and do not support before an answer is drafted.

The tenancy story and retrieval outcomes below are illustrative, not a measured benchmark or client case study. Public legal sources are linked for checking. This is an article about retrieval design, not legal advice.

What is query planning in retrieval-augmented generation?

In retrieval-augmented generation (RAG), query planning is a step between understanding a request and searching the knowledge base. Instead of treating a long message as a single search, the planner identifies its separate questions, relevant dates and required evidence. It then prepares a bounded set of retrieval queries.

Query rewriting makes a query clearer. Query expansion adds useful formulations or vocabulary. Query decomposition separates distinct information needs. A narrative legal question may benefit from all three, but they should preserve the user's meaning rather than introduce new assumptions.

Our article on hybrid search and reciprocal rank fusion for legal RAG covers the retrieval layer: keyword search and semantic search run together, their results are fused, and a reranker can refine the shortlist. Planning sits in front of that layer. It is not a replacement for it.

Why can narrative questions weaken embedding retrieval?

A common dense retriever encodes the query into one vector. That representation must capture the useful meaning of the input, including several topics when the input is a story. Its nearest passages may match the general situation without supplying evidence for every sub-question.

• Several issues compete for one representation. Payment, property damage, a missing document and an unresponsive landlord are related, but the evidence for each may live in a different passage. A single ranking can favour one issue over the others.
• The information need can be implicit. "He never sent me anything about where it was held" may need to be searched alongside the source term "prescribed information". Semantic retrieval can bridge that gap, but should be tested rather than assumed to do so reliably.
• Identifiers need explicit handling. Dates, amounts, clause numbers and names should be retained as structured facts or exact search terms when they affect the answer. A semantic similarity score is not a reliable substitute for an exact constraint.
• Hybrid retrieval still needs a useful query. BM25 may recover exact terms that a vector search misses. However, ordinary narrative wording may not contain the legal terminology needed for a separate issue. Fusion merges ranked lists; it does not itself decide which questions the story contains.

An embedding is not necessarily an arithmetic average of topics, and token counts do not translate into proportional influence on its vector. Narrative questions do not always fail either. The outcome depends on the embedding model, corpus, query processing and retrieval settings. The practical risk is incomplete evidence coverage, not a universal mathematical rule.

Conversation history can add further ambiguity. Concatenating every previous turn into a new search query may retain obsolete facts or an earlier topic. A planner should carry forward relevant context while resolving corrections and follow-up references, rather than blindly searching the whole transcript.

The symptom is a fluent answer that cites real passages but addresses only one of the user's questions. A citation-presence check will not catch that omission. The system must also check whether it answered the request.

A legal RAG example: a tenancy deposit story

The following fictional scenario concerns a private tenancy in England beginning in 2024. The applicable tenancy rules and their versions must be checked for the relevant dates; this example does not determine the tenant's legal entitlement.

I moved into a flat in January 2024 and paid a £1,200 deposit. The landlord never sent me anything about where the deposit was held. In June 2024 he said he would keep £400 for a damaged sofa, but the sofa was already damaged when I moved in and I have photos from the first day. I moved out in September 2024 and he has stopped replying to messages. Can I get my deposit back, and can I claim anything on top because he never protected it?

There are two main requested outcomes: recovering the deposit and investigating an additional remedy. The story also contains evidence about a disputed deduction, missing information and an unresponsive landlord. Importantly, not receiving scheme information does not by itself prove that the deposit was unprotected. A safe plan searches for how to check protection rather than turning the user's suspicion into a finding.

Single-query retrieval: a possible failure pattern

Suppose the baseline embeds the full message, runs hybrid search and selects five passages. An illustrative result set might include a deposit overview, a passage about deposit amounts, tenancy-ending guidance and a dispute summary. These are related to the story, but may omit the evidence needed to address a protection breach.

The draft could then discuss deductions without answering the additional-remedy question. This is a failure mode to test, not an observed result claimed for a particular model or knowledge base. A strong single-query baseline may perform well, so it must be included in any evaluation.

Query decomposition: four focused evidence needs

A planner could produce the following searches. The third column describes evidence to seek, not passages that we claim to have retrieved in a test.

Each query has a traceable purpose. Keyword retrieval can now search for source vocabulary such as "prescribed information", while semantic retrieval searches for related explanations. Hybrid fusion and reranking operate on each evidence need, and duplicate passages can be removed before drafting.

If the corpus supports deposit recovery but not the additional remedy, the output should say so. It should not silently drop the second question or treat missing retrieval results as proof that no remedy exists.

When should a RAG system plan before searching?

Planning adds work, latency and another opportunity for model error. A focused lookup such as "What information must a landlord give about a deposit?" may go straight to retrieval. A message describing several events and asking for several outcomes is a better candidate for decomposition.

Useful routing signals include multiple requested outcomes, several events or dates, unresolved references to earlier turns, and explicit requests to compare sources. Length alone is not enough: a short question can need several documents, and a long quotation may need only one lookup.

A classifier or rule-based gate can make the routing decision. Measure its overhead too: even a gate that skips planning may still add latency. If the gate sends a complex question straight to retrieval, the subsequent coverage check may detect the omission, but that check can fail as well.

Two constraints keep the planner focused:

• Trace every search to the request. Store the user statement or explicit information need that justifies each query. If an inferred legal issue is uncertain, mark it as an assumption or ask for clarification instead of silently adding it as fact.
• Cap the work. For example, allow up to four focused searches, one recovery round and a fixed evidence budget. Four is a design choice, not a universal optimum. Requests exceeding the budget should be narrowed or explicitly returned as partial.

A bounded search-planning and evidence-coverage workflow

The supplied implementation diagram names the planning and drafting component Luna and the retrieval component Masih. Its database label records a snapshot of 85,807 shared chunks. That implementation-specific count is not a benchmark result, a quality score or a claim that every chunk is accessible to every user.

The steps matter more than the component names. This is the proposed control flow, not evidence that a model will follow every instruction correctly.

1. Check evidence coverage per question. Assess each evidence need as supported, partial or missing. "Supported" must refer to the requested fact and its conditions, not merely to a passage on the same topic. Keep the relevant source references for review.
2. Allow one recovery round, then stop. Refine missing or partial searches using what the first retrieval revealed. In the illustrated design, recovery is bounded to one round. If support remains inadequate, preserve that gap in the answer.
3. Select passages for coverage. Deduplicate results and choose evidence across the planned questions, not only the highest similarity scores. The diagram caps selected sources at eight; a separate context-token budget is still required because source lengths differ.
4. Draft with citations, or quote when appropriate. Instruct the drafting model to use the selected evidence, attach source references and distinguish facts from assumptions. If the user requests exact statutory wording, use the verified source text and version rather than presenting a paraphrase as a quotation. Access and reuse restrictions still apply.
5. Check citations and allow one repair pass. Check that cited passages support the associated claims, that source identifiers actually came from retrieval and that the answer is internally consistent. Independent model checks can still be wrong. Remove or qualify unsupported material that remains after repair; human review is needed for consequential use.

The output may be complete, partial with explicit gaps, or a statement that the available sources are insufficient. An inability to answer safely is useful information, especially in legal document processing and cited search.

Retry caps alone do not impose a fixed cost or latency ceiling. A bounded implementation also needs limits on query count, input and output tokens, retrieval candidates, concurrency and total execution time. If a timeout is reached, return a clear status instead of silently extending the loop.

What query planning does not solve

Retrieval quality for each sub-question

A clear plan can still feed a weak retriever. Chunk boundaries, hybrid search, fusion and reranking must be evaluated on the actual documents. Planning may simply produce several misses instead of one. The legal hybrid-search article explains these complementary controls.

Temporal validity and changing law

The right topic in the wrong version can still produce the wrong answer. Retain source versions, effective dates and jurisdiction metadata, and apply the relevant filters to each search. A date in the user's story is a fact to interpret, not automatically the effective date of every legal rule. Ask for clarification where that distinction changes the answer.

Planner drift and prompt injection

A planner can introduce an issue the user never raised, overinterpret a missing detail or mistake quoted material for instructions. Keep user and retrieved text separate from system instructions, restrict tool access, and treat retrieved passages as evidence rather than authority to change the workflow. Showing concise planned searches can make drift visible without exposing hidden model reasoning.

Permissions and source boundaries

Multiple queries create multiple chances to cross an access boundary. Enforce permissions within every retrieval and recovery call, before unauthorised content enters the candidate pool. Source-completion or neighbouring-passage expansion must preserve those same controls. Planning must not bypass the security and governance rules used by direct search.

How to evaluate RAG query decomposition fairly

Build a reviewed set of identifier lookups, conceptual questions and narrative questions. For each narrative, have a suitably qualified reviewer mark the separate information needs, acceptable evidence and important conditions. Around 50 varied stories can be a practical starting set, not a statistically justified minimum or a sufficient release criterion on its own.

Compare a credible single-query baseline with the planned version using the same corpus snapshot, permission rules and source versions. Report the embedding and generation models, retrieval settings and test date. A study of question decomposition for RAG by Ammann, Golde and Akbik examines decomposition and reranking on multi-hop benchmarks. Its results motivate testing the approach; they are not validation of this tenancy example or this implementation.

• Planner precision and recall. Match planned evidence needs to the reviewer's annotations, allowing equivalent wording. Precision measures how many planned searches correspond to a justified need; recall measures how many required needs were covered. Record extra assumptions separately.
• Retrieval hit rate per question. Record whether acceptable evidence appears in the candidate shortlist for each need. Compare with retrieving from the full narrative, and break results down by question type. A hit is not the same as complete evidence when several passages are required.
• Equal-budget and operational comparisons. Four searches returning five passages each have a larger candidate budget than one search returning five. Include an equal-total-candidate comparison, keep final context-token limits comparable, and report the extra calls and reranking cost. Otherwise an apparent planning gain may partly be a larger search budget.
• Coverage-check accuracy. Compare supported, partial and missing labels with human judgements. Report false-supported cases explicitly: a passage that sounds relevant but does not support the claim can prevent a needed recovery attempt.
• Answer completeness and citation support. Score each requested outcome, its supporting evidence and any disclosed gap. An answer that addresses one of two requested outcomes should not receive full credit merely because it is fluent.
• Latency and cost. Measure the whole pipeline, including routing, planning, retrieval, recovery, drafting and checking. Split simple lookups from narratives and report latency percentiles, not only an average. Also test timeouts, refusals and permission-denied cases.

Re-run the evaluation when prompts, models, embeddings, chunking or the corpus change. Keep difficult and failed examples, and avoid publishing a single improvement percentage without its baseline, sample size and budget. This article reports a design and an evaluation method; it does not report measured performance gains.

What buyers should ask in a legal RAG demonstration

Do not test only tidy questions. Supply a realistic, anonymised story containing several facts and at least two requested outcomes. Then ask:

• Which evidence needs did the system identify, and are they traceable to what the user asked?
• Do the selected passages support each outcome, including relevant dates and exceptions?
• Does the answer name missing support rather than fill the gap?
• Can the system show its search plan and source references, without revealing private data or hidden reasoning?
• What happens after a failed search, an invalid citation or an execution timeout?

A narrative query is not an unusual edge case. It is how many people describe document problems. RAG development should therefore evaluate how the system turns that description into evidence needs, not only how well it answers a prepared search phrase.

Related reading

• Why legal RAG needs hybrid search: the keyword, semantic and rank-fusion layer used by each planned search.
• Chatbot, RAG, or RAG with MCP tools?: where retrieval ends and controlled tool use begins.
• Intelligent document search with RAG and OCR: connecting retrieval to document workflows.
• All BlueMouse.ai Insights.

Sources and example notes

• Reading Borough Council: deposit protection schemes. Public background on registration and checking an approved scheme.
• GOV.UK: information landlords must give tenants. Public guidance on the information requirements.
• GOV.UK: if your landlord does not protect your deposit. Public guidance on checking protection and possible court remedies.
• GOV.UK: deposit disputes and problems. Distinguishes scheme dispute resolution from possible court recovery.
• Ammann, Golde and Akbik: Question Decomposition for Retrieval-Augmented Generation, 2025. Research on decomposition and reranking for multi-hop retrieval.
• Haystack documentation: advanced RAG techniques. Implementation resources for query decomposition and related retrieval approaches.

Sources checked on 5 November 2026. The story, search plan and possible retrieval outcomes are illustrative and contain no client data. The supplied workflow diagram documents an implementation design, not a comparative test. Legal applicability depends on facts, dates and jurisdiction; obtain qualified advice before acting on a legal conclusion.