Monday AI Insight. A user types "why is it doing that again?" into your RAG chatbot. Your retriever embeds those six words, finds the nearest chunks, and hands the model a pile of irrelevant text. The model then answers confidently from the wrong context.
The lesson: the user's question is usually a bad search query. Rewrite it before you retrieve.
Users write questions. Your documents contain answers. Those look different: questions are short, vague, full of pronouns and missing context ("it", "that error", "the second option"), while the passages you want are longer, specific, and written in the vocabulary of your docs. Embedding the raw question and hoping it lands near the right chunk works for easy queries and fails quietly on the rest.
Three rewrites that fix most of it
1. Condense follow-ups into a standalone query
In a chat, the second and third turns are rarely searchable on their own. "What about on Postgres?" means nothing without the previous turn. Before retrieval, ask a cheap model to turn the chat history plus the latest message into one standalone question:
History: "How do I add a vector index in pgvector?" / Latest: "what about for cosine distance?"
Standalone: "How do I create a pgvector index for cosine distance?"
This one step usually matters most, because follow-ups are where retrieval breaks most visibly.
2. Expand into a few query variants (multi-query)
One phrasing can miss documents that use different words. Generate 2–4 rephrasings (synonyms, the product's own terminology, a more specific version), retrieve for each, and merge the results. Reciprocal Rank Fusion (RRF) is a simple way to merge ranked lists without tuning score scales. Keep the original query as one of the variants so you never do worse than the baseline on queries that were already fine.
3. HyDE: search with a hypothetical answer
HyDE (Hypothetical Document Embeddings) comes from Precise Zero-Shot Dense Retrieval without Relevance Labels (Gao, Ma, Lin, Callan, CMU and Waterloo, 2022). The idea:
- Ask an instruction-following LLM to write a hypothetical document that answers the query.
- Embed that document, not the question.
- Use its vector to find similar real documents in your corpus.
The generated document may contain false details, and that's fine: as the paper puts it, the encoder's dense bottleneck filters out incorrect details, and retrieval grounds the result in your actual corpus. You never show the hypothetical text to the user. The authors report HyDE significantly outperforms the unsupervised dense retriever Contriever and performs comparably to fine-tuned retrievers across tasks like web search, QA and fact verification, and in several languages.
Why it works: an answer-shaped passage sits closer, in embedding space, to the answer passages you stored than a short question does.
The broader pattern: Rewrite-Retrieve-Read
Query Rewriting for Retrieval-Augmented Large Language Models (Ma et al., 2023) frames this as Rewrite-Retrieve-Read instead of the usual retrieve-then-read: there is "inevitably a gap between the input text and the needed knowledge in retrieval," so you adapt the query itself. They first prompt an LLM to rewrite the query, then go further and train a small rewriter with reinforcement learning using feedback from the LLM reader, reporting consistent improvements on open-domain and multiple-choice QA. You don't need the RL part to benefit; a prompted rewrite is the starting point.
A minimal implementation
REWRITE_PROMPT = """Rewrite the user's latest message as a standalone search query using the conversation for context. Then give 2 alternative phrasings that use terms likely to appear in technical docs. Return JSON: {"standalone": "...", "variants": ["...", "..."]}""" def retrieve(history, message, k=8): q = llm_json(REWRITE_PROMPT, history=history, message=message) queries = [message, q["standalone"], *q["variants"]] # optional HyDE: embed a short hypothetical answer as one more query hypo = llm_text(f"Write a short passage that answers: {q['standalone']}") ranked_lists = [vector_search(embed(text), k=k) for text in queries + [hypo]] return reciprocal_rank_fusion(ranked_lists)[:k] # merge, keep top k llm_json, llm_text, embed, vector_search and reciprocal_rank_fusion are placeholders for your own stack.
The trade-offs (don't skip these)
- Latency and cost. Each rewrite is an extra LLM call before retrieval even starts. Use a small, fast model, and skip rewriting for first-turn queries that are already specific (an ID, an error code, a quoted phrase).
- Drift. A rewriter can "helpfully" change what the user asked. Keep the original query in the mix and log the rewrite next to it so you can see when it goes wrong.
- HyDE needs a model that knows the domain a bit. If the LLM has no idea what your internal product is, its hypothetical answer may point in the wrong direction. Test it on your own queries.
- Exact matches still need keyword search. Rewrites don't replace BM25 for IDs and error codes; combine them.
How to know it helped
Build a small set of real user queries (including messy follow-ups) with the chunks that should come back, and compare recall@k with and without each rewrite step. Turn on only the steps that move the number for your data.
Checklist
- Condense chat follow-ups into standalone queries before retrieval
- Add 2–4 query variants and merge results with RRF
- Try HyDE for short or vague questions; never show the hypothetical text
- Always keep the original query as one of the searches
- Use a small model for rewrites and skip them for already-specific queries
- Log rewrites and measure recall@k on real queries
Where does your RAG retrieval break most: first questions, or follow-ups?
References
- Gao, Ma, Lin, Callan (2022). Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE). https://arxiv.org/abs/2212.10496
- HyDE code: https://github.com/texttron/hyde
- Ma, Gong, He, Zhao, Duan (2023). Query Rewriting for Retrieval-Augmented Large Language Models. https://arxiv.org/abs/2305.14283
- Rewrite-Retrieve-Read code: https://github.com/xbmxb/RAG-query-rewriting
- Cormack, Clarke, Büttcher (2009). Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods. https://dl.acm.org/doi/10.1145/1571941.1572114
