Spider, BIRD, LiveSQLBench. If you have evaluated a text-to-SQL system in the last five years you have quoted a number from one of them. All three ask the same question: given a schema and an English question, does the system produce SQL that returns the right rows? None of them ask who is asking. Every score you have seen was produced by a system with unrestricted read access to the entire database. That is not how anyone runs one in production, and a paper accepted to SIGMOD 2027 has now measured what happens when you close the gap. The paper Benchmarking Text-to-SQL under Role-Based Access Control, by Yang Fei, Yangfan Jiang, Yin Yang and Xiaokui Xiao (arXiv, July 2026). They take the three benchmarks above and add what production has and benchmarks don't: roles, and policies attached to them. The scale of the augmentation: 53 databases, 399 tables, 3,353 columns 21,502 role-annotated query instances policies at column-operation granularity — not "can this role read this table" but "can this role SELECT this column" roles synthesised per database by an LLM-assisted pipeline, including scoped administrator roles Then they score existing systems against it. From the abstract: many high-performing systems, open-weight LLMs especially, "show sharp performance degradation once access constraints are in place, due to frequent RBAC violations." I want to be careful here, because I have not run this myself and the benchmark is not released yet — the paper says the pipeline, toolkit and datasets are coming to a public repository. What follows is the mechanism behind that sentence, which is the part I do have direct experience of. The metric that matters The paper's sharpest contribution is not the dataset. It's a category of failure their metrics are built to expose, which they call RBAC-rejected successes: SQL that is judged correct under normal evaluation and violates the access policy. Sit with that for a second, because it is the whole problem in five words. Under Spider's or BIRD's grading, that query passes. It answers the question. It returns the right rows. It is also a query the person who asked was never permitted to run. Your evaluation harness scored it as a win. This is why the degradation is sharp rather than gradual. It isn't that access control makes the SQL-writing task harder. It's that a metric which never looked at authorisation was quietly counting violations as successes, and once you look, they move columns. Why this happens, architecturally Here is the part I have spent this year on, and the reason the paper's result did not surprise me. Every NL2SQL stack has a schema-selection step. It has to: a real schema is hundreds or thousands of objects and you cannot put all of it in a prompt. So something narrows the candidates before the model writes anything. question ──► select tables ──► model writes SQL ──► execute ↑ RLS / VPD / grants act HERE Your database's access control is at the far right. It acts on execution. But selection happened at the far left, and it was blind — a ranker matching words against a catalogue, with no idea who is asking. So the model gets handed hr_compensation because a support agent's question happened to score near it. The model does its job perfectly and writes correct SQL against the table it was shown. Then row-level security does its job perfectly and filters every row. And the user sees: No records found. Which is a wrong answer wearing the costume of an empty result. Nothing errored. Nothing was logged as a denial. The agent will say "there are no compensation records matching that" with total confidence, and the person reading it has no way to distinguish "the data doesn't exist" from "you aren't allowed to see it." Those are very different sentences and your stack just collapsed them into one. The variant that should worry you more: the schema itself is information. A table named hr_compensation_2027_layoffs tells the reader something real, even when zero rows come back. Putting that name in a prompt is a disclosure, and RLS cannot retract it, because RLS filters rows and the leak was in the DDL. What the paper does not do It does not propose a system. It is a benchmark and an evaluation methodology, deliberately. Read that as good news: a top-tier venue has named and measured the problem, and left the engineering open. The obvious fix — "just re-check permissions after the SQL comes back" — doesn't hold, for two reasons. It's too late for the DDL disclosure above. And a post-hoc check can only tell you the query was disallowed; it cannot produce the answer the caller was entitled to, because the objects that would have answered it were never candidates. The fix has to be at selection. Whatever narrows hundreds of objects to six has to know who is asking: sel = catalog.select( "salary by employee", principal=Principal("okta:jdoe", roles={"analyst"}), ) sel.table_names # hr_compensation is not here sel.prompt_fragment() # and its name is not in the text the model sees Not ranked last. Absent. The difference matters: a table ranked last is still in the candidate set, still one prompt-budget change away from being included, and still named in your logs. Two properties worth insisting on if you build this yourself: A restricted object and a nonexistent one must be indistinguishable. Otherwise "access denied" versus "no such table" leaks the schema one probe at a time. Scoping is not authentication. If your selector takes a principal, then whatever hands it that principal is now a trust boundary. A component that accepts principal="okta:admin" from an untrusted caller has given away everything. Scoping decides what an authenticated identity may see; it does not establish the identity. What I'd like to be able to tell you I maintain a library that does exactly the selection step above, and the honest position is that I have self-reported numbers and no third-party ones. That is worth precisely as much as you'd expect. This benchmark is the fix for that, and it's the reason I'm writing about a paper rather than a release. When the authors publish the pipeline, I will run it and post the results — including if they are bad, which is a promise that costs nothing to make and something to keep, so hold me to it. Until then, the useful takeaway isn't about any library. It's this: if you are evaluating a text-to-SQL system, your benchmark score was measured with God-mode access to the database, and the number you are about to put in a slide does not describe how the system behaves for your actual users. The gap between those two things has now been measured by people with no product to sell, and it is large. The paper: Benchmarking Text-to-SQL under Role-Based Access Control, Fei, Jiang, Yang & Xiao, SIGMOD 2027. The base benchmarks: BIRD, LiveSQLBench, Spider. The selection library is schemagate, Apache-2.0, with a browser demo at ashishsinha1602.github.io/schemagate that runs the real selector client-side — flip the caller's roles and watch the restricted table leave the prompt.