The Registry That Wasn't Enough: How We Went from "Knowing APIs" to Actually Verifying Against PyPI

The Registry That Wasn't Enough: How We Went from "Knowing APIs" to Actually Verifying Against PyPI

Continuation of the previous article on detecting hallucinations in AI-generated code.

In the previous article, I told the story of how this idea was born: a requests.get_json() that never existed, that passed tests, linter, and type checker, and that would only have failed in production. The tool I built back then relied on a handcrafted registry of valid APIs: a map of library → known functions, and a simple rule: if the library is in the registry and the function isn't, it's a hallucination. It worked for requests, json, and os, and it detected nonexistent packages by querying PyPI.

But the registry had an obvious limitation: it only judged what it knew. And as soon as you stepped outside requests, json, and os, the tool went silent. Not because there were no hallucinations, but because it had nothing to compare against. That ceiling is what I've been thinking about for the past few weeks. And this is what came next.

Of course, this work doesn't start from scratch. Recent research confirms that library hallucinations are systematic (5.2% of packages mentioned by commercial models don't exist, and up to 21.7% for open-source models), that static analysis has a ceiling of 48.5% to 77% of what it can possibly detect, and that verification against real documentation significantly reduces false positives. Our project addresses some of those limitations (invented packages, APIs of real packages verified against wheels) and leaves others documented (APIs of nonexistent packages, advanced slopsquatting). This continuation covers what we've done and what remains.
How to Verify More Rigorously?

The question that changed everything was: what if, instead of maintaining a hand-written list of valid functions, we read what the package actually contains?

PyPI doesn't just serve packages: it serves wheels, which are ZIP files containing the code and — very importantly — the type stubs (.pyi) when the package includes them. If you download the wheel and open it, you can check whether class DataFrame exists in pandas without installing anything and without executing any of the user's code.

That's exactly what we did. And it changes the game: you no longer depend on someone maintaining a list of functions by hand. You depend on what the package actually publishes on PyPI. Coverage stops being artisanal and becomes that of the ecosystem.
The Cache, or How Not to Punish PyPI on Every Analysis

The first practical problem was obvious: if every analysis queries PyPI for every import, on a slow network like mine, the analysis drags on forever. And if PyPI goes down, the detector just sits there waiting. So we built a two-level cache (memory + disk) with TTLs:

exists (does the package exist on PyPI?): 7 days. Reputation metadata (first release, maintainers): 30 days. Network errors: 10 minutes (to avoid retrying in a loop when the network is flaky). 

And a design decision that was already there in the previous article but now becomes central: fail-open. If we can't verify (timeout, PyPI down), we don't report the package as nonexistent. Better an occasional false negative than a false positive that erodes trust.

The result is noticeable: the second run on the same file goes from ~50 seconds to 17 milliseconds. And all of that without touching the normal flow: if you don't configure the cache, the detector works exactly as before.
Canonicalization, stdlib, and the False Positive That Taught Us Something

With the cache came the fine details. PyPI normalizes names (PIL is installed as Pillow, sklearn as scikit-learn, cv2 as opencv-python), and imports bring submodules (tensorflow.keras, os.path). If you don't canonicalize, the cache misses and odd false positives start appearing.

One of the most instructive: imp (a stdlib module, deprecated) was flagged as typosquatting of pip. The edit distance was 1, and the heuristic didn't know that imp is stdlib. The lesson: stdlib lists and PyPI package lists must stay in sync, or the detector starts inventing problems where there are none. Today imp, distutils, and friends are recognized as stdlib and are not flagged.
Actually Verifying Attributes: pandas.DataFrame Is No Longer a False Positive

Here's the biggest change compared to the previous article. Before, api-hallucination was heuristic: it looked for suspicious names, it didn't verify. And that produced false positives like pandas.DataFrame, which does exist but wasn't in the registry.

Now there's a deep mode (deep) that, when you enable it:

Queries PyPI for the package. Downloads the wheel (once, cached per version). Opens the ZIP and searches for the attribute in all the package's .pyi and .py files. Returns one of three states: confirmed, not found, or unverifiable. 

The result, measured with the same test file: 15 findings without deep mode, 14 with deep mode. The difference is exactly the false positive for pandas.DataFrame, which disappears. And more importantly: pandas.funcion_inventada is detected, with evidence that it's not in the real wheel.

That's something the handcrafted registry could never provide. And we did it without installing Python on the server, without executing the user's code, and without adding third-party dependencies: just HTTP, ZIP, and a bit of regex.
IP Control, or How to Open It Up Without Getting Your Server Knocked Over

Deep mode downloads wheels. That costs bandwidth and time. Before opening it to the public, we needed a limit. So we added IP-based control: 3 deep analyses per day per IP, with a daily reset.

Three details that mattered to us:

Localhost is exempt, so development tests don't stop because they hit the limit. Otherwise you'd lock yourself out of your own machine. IPs can be simulated in development with X-Forwarded-For and a DETECTOR_DEV=1 variable. That way you can test the limit without deploying. In production, DETECTOR_TRUST_PROXY=1 to read the real IP behind a proxy. Otherwise all users would share the proxy's IP. 

The limit is simple, but it's the difference between opening deep mode with a safety net and opening it without one.
The Four Test Files, and What They Actually Tell Us

To measure without fooling ourselves, we wrote four files:
File What it measures Result
test_debe_detectar.py Recall (catch the bad stuff) 14–15 findings, almost all correct
test_no_debe_detectar.py Precision (don't flag the good stuff) 0 findings ✅
test_limitaciones.py Known false negatives/positives 1 finding (a true positive by heuristic)
test_syntax_error.py Broken syntax 0, "file looks clean" ❌

The good part: precision is perfect on clean code. The honest part: there's a UX bug with broken syntax (the detector says "clean" when the file doesn't even compile) and there are structural false negatives (dynamic imports, from-import of names, nonexistent submodules) that require deeper analysis. We have them documented, not hidden.
What's Next

So the project is no longer "an API registry." It's a detector that actually verifies against PyPI, with caching, opt-in deep mode, and usage control. What's left to review from this milestone:

Wheel download events in the progress bar (today, with deep mode, the bar freezes while the wheel downloads). --deep in the CLI, so the terminal has the same power as the web. Reporting syntax-error as a finding, instead of saying "file looks clean". Evidence and status on each finding (confirmed, probable, unverifiable). pyright/mypy only as a local option, never on the server (slow, requires installation, and without stubs it flags nothing). 

And there's one more thing we'd like to mention. Once we have hosting — complicated from my country under siege — we'll launch a web app that already allows running the analyses online. It's a tool that's already built and working locally: you upload a .py, you see progress in real time (with an SSE progress bar), and you get the findings, including deep mode with verification against PyPI wheels. But for that web app to be sustainable and not fall over at the first sign of traffic, we need a contribution to the tool's development. It's not a payment to use the service: it's a way for the project to cover hosting, the bandwidth for wheel downloads, and maintenance time. The idea is that anyone who wants to use deep mode intensively can contribute, and anyone who just wants to try basic analysis can do so without barriers. The code will remain open; the hosted web app will be how the project sustains itself. We'll share more details when it's ready.

The previous article ended with an idea: "the linter sees what it knows; the type checker sees what's in the signature; neither knows whether the method actually exists." This is an attempt to close that gap. Not with a bigger list, but by reading what the package publishes. It's still not perfect: there are false negatives we acknowledge and a syntax bug pending. But the difference between "there's no hallucination" and "it couldn't be verified" is no longer hidden: we measure it, we document it, and we try to reduce it.

If you try deep mode and find a false positive or one that slips through, I'd love to hear about it. This project gets corrected by what others find.
References

Spracklen, J., Wijewickrama, R., Sakib, A.H.M.N., Maiti, A., & Viswanath, B. (2025). We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs. 34th USENIX Security Symposium. arXiv:2406.10279. Miranda-Pena, C., Reeson, A., Paris, C., Poon, J., & Kummerfeld, J.K. (2026). An Empirical Analysis of Static Analysis Methods for Detection and Mitigation of Code Library Hallucinations. arXiv:2604.07755. Tileria, M., et al. (2026). Hallucination Inspector: A Fact-Checking Judge for API Migration. arXiv:2604.20202. Raj, A., & (co-author). (2026). Names Can Hurt: Spotting Slopsquatting Risks Caused by Package Name Hallucinations in Local Coding LLMs. arXiv:2608.23897. Churilov, A. (2026). The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort. arXiv:2605.17062.