The cataloguing step is supposed to be the easy win. You have a schema whose tables are called ecm_template_link and v_pmpm, your users ask questions in English, and the gap between those two vocabularies is why retrieval misses. So you point a model at every table, get back a sentence describing each one, index the sentences alongside the names, and now the corpus speaks English too. I did that to a real 1,245-object schema. Recall went down. Not by a little. The table literally called contacts sat at rank 3 for the question "show the contacts of xmagnet" before cataloguing. After cataloguing it was below rank 40 — off the end of anything I would put in a prompt. The descriptions were fine. I read them. They were accurate, specific, and they made the system worse. This post is what was actually happening, why the obvious fix doesn't work, and the one that does. None of it is specific to text-to-SQL. If you are enriching documents before indexing them — summaries, generated titles, hypothetical questions, keyword expansion, anything — the same mechanism is available to bite you, and it will not announce itself. The mechanism, in one line Every description you generate is written in the same vocabulary as every other description, so enrichment raises the document frequency of exactly the words your users type. BM25 scores a term by inverse document frequency: idf(t) = log(1 + (N - n_t + 0.5) / (n_t + 0.5)) N is the corpus size, n_t the number of documents containing the term. A term in few documents is informative and scores high; a term in most documents is worthless and scores near zero. That is the entire point of IDF, and it is normally a good instinct. Now think about what a model writes when you ask it to describe a table in a CRM schema. It writes about contacts. It writes about contacts when describing contacts, and also when describing contact_lists, campaign_recipients, email_events, tenants, users, and the audit table that logs changes to any of them — because in a CRM, almost everything is about contacts in some defensible sense. The descriptions are not wrong. They are correlated. After cataloguing, the token contact appeared in roughly 1,072 of the 1,245 documents — a figure I can reconstruct from the IDF it produced, which was 0.15. For comparison, here is what the same schema gives you for a genuinely rare term: term documents containing it idf contact, after cataloguing ~1,072 of 1,245 0.15 contact, in the name field only 17 of 1,245 4.27 tenant, in the name field only 30 of 1,245 3.71 the 0 of 1,245 — The tokenizer does not strip stopwords, so the and and are in that index too, sitting near zero because they are in everything. At 0.15, contact had joined them. The word the user typed was, for scoring purposes, a function word. The second half, which is worse IDF collapse alone would flatten the ranking. What actively inverted it was length normalisation. The b parameter in BM25 penalises long documents, on the sound theory that a long document containing your term is less about your term than a short one that contains it: score += idf(t) * f * (k1 + 1) / (f + k1 * (1 - b + b * len / avg_len)) Ask which object in a CRM schema has the longest document, and the answer is the central one. contacts in this schema has 55 columns. Add a generated description and a row of alias words and its document is several times the corpus average. Meanwhile contact_import_log has six columns and a one-line description, so it is short, tidy, and — as far as the length prior is concerned — much more about contacts. So the two effects compound in the same direction: IDF collapse removes the signal that would have separated contacts from the forty other tables mentioning contacts. Length normalisation then actively sorts what's left by inverse centrality, because the most important table in a schema is reliably the one with the most columns. Cataloguing didn't add noise. It added correlated noise, and correlated noise attacks the exact query it was meant to help. The questions that degraded most were the ones cataloguing exists to serve — plain English, no schema words. Questions that named a table outright were mostly fine, because they had a rare token to hang on. That is a nasty failure profile: the feature looks fine on your smoke tests and fails on your users. The fix that doesn't work The obvious response is to trust descriptions less. One index, but weight the generated text below the real text. I did this first. It helps a bit and it is the wrong lever, for a reason that took me a while to see: weight and dilution act at different stages. Down-weighting scales the contribution of a term after IDF has already been computed over a corpus the descriptions polluted. contact is still worth 0.15 in the name's own score, because name and description live in one bag of words and IDF is a property of the bag. You have made a bad channel quieter without making the good channel accurate again. And the cost is real. Starving the prose weight cost me "per member per month cost" → v_pmpm, which is the single best example in the whole schema of a question only a description can answer. There is no lexical path from that phrase to that name. The description was the only bridge and I had just defunded it. So: down-weighting trades away the wins to partially mitigate the losses. You end up tuning a scalar that makes both worse than they need to be. The fix that works Score the fields separately and fuse the rankings, rather than concatenating the fields and scoring once. Three BM25 indexes over the same objects: self._bm25 = _BM25([doc.embed_text() for doc in docs]) # everything self._bm25_name = _BM25([_name_text(doc) for doc in docs]) # identifiers self._bm25_prose = _BM25([_prose_text(doc) for doc in docs]) # written text where def _name_text(doc): """Just the identifiers: schema, name, and the name split on underscores.""" return " ".join(x for x in (doc.schema, doc.name, doc.name.replace("_", " ")) if x) def _prose_text(doc): """Everything written *about* the object: hint, description, comments.""" parts = [doc.hint or "", doc.description or ""] parts.extend(c.comment or "" for c in doc.columns) return " ".join(x for x in parts if x) Then fuse by reciprocal rank rather than by score: for q in candidates: s = 0.0 if q in vec_rank: s += vector_weight / (RRF_K + vec_rank[q] + 1) if q in lex_rank: s += lexical_weight / (RRF_K + lex_rank[q] + 1) if q in name_rank: s += NAME_WEIGHT / (RRF_K + name_rank[q] + 1) if q in prose_rank: s += PROSE_WEIGHT / (RRF_K + prose_rank[q] + 1) Both halves of the bug die at once, and it is worth being precise about why, because "just use fielded search" is advice people give without the mechanism: IDF is recomputed per field. In the name index, the only text is identifiers. Nothing a model writes can ever enter it. contact appears in 17 names out of 1,245, so its IDF is 4.27 instead of 0.15 — 28× the discriminating power, restored by construction rather than by tuning. Length is per field too. The name index's document length is the length of the name. contacts is two tokens whatever else you attach to the object. The 55 columns cannot inflate it, so the length prior stops punishing centrality. Fusion is over ranks, not scores. This is the part that contains a bad catalogue, and it's why I could raise the prose weight back to parity. A channel can only ever contribute its own ranking. If a weak model writes "Stores data about users and their settings" about all 1,245 objects, the prose channel becomes uniformly useless — every object ranks the same, the channel contributes nothing that discriminates, and the name and body channels decide the result unchanged. The floor becomes "no better than before cataloguing" instead of "worse than before cataloguing". That last property is the one I actually care about. It means pointing a small local model at your schema is safe. Not good, necessarily — a 1.5B model writes considerably worse descriptions than a frontier model, and I'd rather you use the good one. But safe: bad prose can no longer bury the object it describes, so the downside of trying is bounded. What this generalises to The pattern is not about databases. It is: Generated text about a corpus is written in the corpus's own vocabulary, so enrichment inflates document frequency for the domain's central terms — the ones users search with — and inflates document length most for the items that matter most. Anywhere you generate text and index it next to original text, in the same field, you have signed up for both effects: Summaries prepended to chunks. Every summary in a corpus about Kubernetes says "Kubernetes". Hypothetical-question generation (HyDE-style indexing). You are synthesising the user's own phrasing, at scale, across every document. That is IDF dilution as a product feature. Keyword and synonym expansion. Same shape, more concentrated. LLM-written titles or alt-text merged into the body field. None of these are bad ideas. I still catalogue schemas; recall on business-phrased questions is far better with descriptions than without. The claim is narrower: enrichment belongs in its own field, always. The cost of separating fields is one more index and a fusion step. The cost of not separating them is a regression that shows up only on your most important queries and looks like "retrieval is just hard". If you want to check whether this is happening to you, it is one query and no instrumentation: take the ten nouns your users actually type, and print their document frequency before and after your enrichment step. If any of them are now in more than half your documents, that term is doing nothing, and it was probably doing something before. What it doesn't fix Honesty about the edges, since the above reads tidier than the week did: Fielded scoring does not make a bad catalogue good. It makes it harmless. If your descriptions are generic, you get the pre-cataloguing ranking back, not a better one — which is the right outcome, but don't read it as a licence to skip evaluating the model that writes them. It also introduces a knob per field, and I do not have a principled method for setting them. Mine are all at parity because that tested best across six schemas, not because parity is theoretically correct. And separating fields cannot fix a term that is genuinely common in the names too. A schema with 300 tables actually called contact_something has a real ambiguity problem, and no amount of field isolation invents the information to resolve it. The measurements here come from schemagate, an open-source library (Apache-2.0) that does the retrieval step for text-to-SQL. The relevant code is in catalog.py — the comments around the three _BM25 constructions are where I wrote this down while it was still fresh. There's a browser demo at ashishsinha1602.github.io/schemagate that runs the real selector client-side on six sample schemas, if you'd rather poke at the ranking than read about it. If you run the document-frequency check on your own corpus, I'd like to know what it says — particularly if it says nothing is wrong, because I'd like to know what makes a corpus immune.