Every sentence was cited. The answer was still wrong.
Ask a company AI whether you can refund a customer after last month's exception. It finds the contract, quotes the 30-day rule and says no, with a citation. The exception sits in an email it never searched.
Here's that failure as a script you can run. It has no API key, no model and no embeddings. The refund is a made-up example.
I framed the post as “RAG in 2023 vs 2026”: search once versus check and search again. The dates are shorthand. Anthropic's Contextual Retrieval already combined hybrid search and reranking in 2024.
The script
Save this as rag_loop.py. It uses only Python's standard library, with SQLite's FTS5 enabled for BM25 search. Python 3.9 and 3.15 print the same output.
# Synthetic data. BM25 and the permission filter are real (SQLite FTS5). # The check and the answers are keyword rules, so no model or API is needed. import re import sqlite3 MAX_ROUNDS = 3 DOCS = [ # id, source, who may read it, text ("contract-s4", "contract", "all", "Section 4 Refunds. Customers may request a refund within 30 days of purchase."), ("contract-s9", "contract", "all", "Section 9 Renewals. Renewal notice is due 30 days before the term ends."), ("ticket-118", "ticket", "all", "Acme asked for a refund on day 41. Waiting on finance. No decision yet."), ("email-1012", "email", "finance", "12 October. Finance director approved a one-time refund exception for Acme."), ] db = sqlite3.connect(":memory:") db.execute("CREATE VIRTUAL TABLE docs USING fts5(id UNINDEXED, source UNINDEXED, readers UNINDEXED, text)") db.executemany("INSERT INTO docs VALUES (?, ?, ?, ?)", DOCS) def search(words, source, role, k=2): # The permission filter sits inside the query, so every repeat search gets it too. sql = """SELECT id, text, readers FROM docs WHERE docs MATCH ? AND source = ? AND readers IN ('all', ?) ORDER BY bm25(docs) LIMIT ?""" hits = db.execute(sql, (" OR ".join(words), source, role, k)).fetchall() assert all(readers in ("all", role) for _, _, readers in hits), "permission leak" return hits def proven(hits, proof): # shortcut: a keyword rule stands in for the model. Ask a model here in a real system. return next((doc_id for doc_id, text, _ in hits if all(w in text.lower() for w in proof)), None) def run(title, customer, role, loop=True): print(f"\n{title}\n customer={customer} role={role}") needs = { # sources to try in order, search words, words that count as proof "rule": (["contract"], ["refund", "days"], ["refund", "30 days"]), "exception": (["ticket", "email", "drive"], ["refund", "exception", customer], ["approved", customer.lower()]), } queues = {need: list(spec[0]) for need, spec in needs.items()} if not loop: queues["exception"] = [] # The single search never looks for the exception. limit = MAX_ROUNDS if loop else 1 evidence, seen, used = {}, set(), 0 for used in range(1, limit + 1): for need, (_, words, proof) in needs.items(): if need in evidence or not queues[need]: continue source = queues[need].pop(0) hits = search(words, source, role) seen.update(hit[0] for hit in hits) print(f" round {used}: {need} -> {source}: {[hit[0] for hit in hits] or 'nothing'}") if found := proven(hits, proof): evidence[need] = found if len(evidence) == len(needs): break missing = [need for need in needs if need not in evidence] if not missing: reply = f"Yes. {customer} has an approved exception to the 30-day limit. [{evidence['rule']}] [{evidence['exception']}]" elif loop: reply = f"Partly. The limit is 30 days. [{evidence['rule']}] Could not confirm: {', '.join(missing)}." else: reply = f"No. Refunds end 30 days after purchase. [{evidence['rule']}]" cited = all(doc_id in seen for doc_id in re.findall(r"\[([\w-]+)\]", reply)) print(f" answer: {reply}") print(f" cited: {'yes' if cited else 'no'} sufficient: {'no' if missing else 'yes'} rounds: {used}/{limit}") run("A. Search once", "Acme", "finance", loop=False) run("B. Loop", "Acme", "finance") run("C. Loop, no permission to read that email", "Acme", "support") run("D. Loop, no exception exists", "Globex", "finance") Run it:
python3 rag_loop.py You get this:
A. Search once customer=Acme role=finance round 1: rule -> contract: ['contract-s4', 'contract-s9'] answer: No. Refunds end 30 days after purchase. [contract-s4] cited: yes sufficient: no rounds: 1/1 B. Loop customer=Acme role=finance round 1: rule -> contract: ['contract-s4', 'contract-s9'] round 1: exception -> ticket: ['ticket-118'] round 2: exception -> email: ['email-1012'] answer: Yes. Acme has an approved exception to the 30-day limit. [contract-s4] [email-1012] cited: yes sufficient: yes rounds: 2/3 C. Loop, no permission to read that email customer=Acme role=support round 1: rule -> contract: ['contract-s4', 'contract-s9'] round 1: exception -> ticket: ['ticket-118'] round 2: exception -> email: nothing round 3: exception -> drive: nothing answer: Partly. The limit is 30 days. [contract-s4] Could not confirm: exception. cited: yes sufficient: no rounds: 3/3 D. Loop, no exception exists customer=Globex role=finance round 1: rule -> contract: ['contract-s4', 'contract-s9'] round 1: exception -> ticket: ['ticket-118'] round 2: exception -> email: ['email-1012'] round 3: exception -> drive: nothing answer: Partly. The limit is 30 days. [contract-s4] Could not confirm: exception. cited: yes sufficient: no rounds: 3/3 The search and permission filter execute in SQLite. Source order, sufficiency and answers are fixed rules. This reproduces control flow, not retrieval quality. cited: yes only checks that cited IDs appeared in retrieved results; it doesn't check whether they support the answer.
What each run tests
A: missing source. cited: yes and sufficient: no on the same line. A citation doesn't prove the answer is complete. A reranker won't help, because it scores the documents you give it. It can't add the missing email.
B: bounded repeat. The ticket mentions the refund but holds no decision. So the next round searches email, finds it and stops at round 2 of 3.
C: permission limit. The filter is inside the SQL, so the repeat search gets it too. The email never comes back, and the answer names what's missing. The role is a fixture; in production, derive it from authenticated identity. Enforce access at the search, and again when a source document is opened.
D: no answer exists. Search returns the Acme email for Globex. The check doesn't count it. The for loop ends at the cap, so the question can't spin forever. A real loop also needs time and token limits, enforced in code.
Now break it
Add this line to DOCS, after the approval email:
("email-1020", "email", "finance", "20 October. Finance director withdrew the Acme refund exception."), Run B now prints this:
round 2: exception -> email: ['email-1020', 'email-1012'] answer: Yes. Acme has an approved exception to the 30-day limit. [contract-s4] [email-1012] cited: yes sufficient: yes rounds: 2/3 The withdrawal ranks first. The answer still says yes. Every sentence is cited, and the answer is wrong again.
The check only asks whether an approval exists. A real one also asks whether it applies, whether that person could approve it, and whether it still stands. So test on conflicting versions.
Where this stops
The script exercises keyword search, a deliberately weak sufficiency check, and a round cap. It doesn't test vector retrieval, reranking or exact-ID matching. The full design adds contextual chunks, query planning, hybrid search and reranking. Google's agentic RAG framework treats sufficient context as its own check.
For a real benchmark, keep the synthetic cases and replace the fixed routing, keyword checks and canned answers. Compare vector-only retrieval, hybrid, hybrid plus reranking, and a capped loop. Log how much required evidence reaches the top results, answer correctness, tokens and wall time.
I wouldn't build the whole loop on day one. A simple lookup should skip it. Start with hybrid search and a reranker, measure on real questions, and add planning or a second round only where that test set fails.
The maintained article, with all six steps and the models I'd benchmark for each, is on argbe.tech.
