What the sites behind the 400 most popular npm packages actually do

Every number below is from the first published run of the corpus (CLI 0.0.54, 2026-11-03) and can be rebuilt from the corpus's README. The live table, refreshed every Monday, is at rankcli.dev/data/npm-sites.

Every week a workflow takes the homepages of the 400 most popular npm packages, by npm's own
popularity order, and fetches each one once as a browser, once as each of eleven crawlers,
plus its robots.txt and sitemap. It keeps no content, only what it observed. This is the
first run: 367 of the 400 sites answered (the rest are GitHub Pages that moved, one 429, one
403 to everyone).

The numbers

Median score 70 out of 100. Frameworks the audit recognised: 84 Next.js, 51 Vue, 38
Docusaurus, 24 Astro, 17 React, 5 Nuxt, 5 WordPress; 121 it could not name.

Three things worth saying

Four in ten have no sitemap. These are the documentation sites of the libraries half the
web is built with, and 150 of them give a crawler no list of pages. Crawlers find pages by
following links; a site that ships a navigation in JavaScript and no sitemap is discovered
one page at a time, if at all. The framework-detected sites are not better: Next.js sites
are missing a sitemap about as often as the rest.

Half publish an llms.txt, and the share tracks the tools people use to build agents.
174 sites have one: 54 of the 84 Next.js sites (64 percent), 15 of 24 Astro sites (63
percent), 29 of 51 Vue sites, 20 of 38 Docusaurus sites. No search or answer engine reads
the file; Google says it neither helps nor harms, and in a 137,000-domain study 97 percent of
these files received no request in a month. What the file is for is coding agents reading
documentation, and the people who build those agents are the people publishing it. That is a
convention, not a ranking lever, and the number is a measure of who is in the corpus.

Blocking is rare, and some of it looks accidental. Eleven sites refuse the training
crawlers at the edge and eleven disallow them in robots.txt; that is the deliberate kind, and
it costs nothing a user sees. The other kind is smaller and stranger. Three sites refuse
Googlebot or bingbot while a browser gets 200: ag-grid.com answers both with a 403, mths.be
refuses bingbot, and lokalise.github.io serves a challenge page to Googlebot, bingbot and the
three AI search crawlers alike, which a crawler cannot solve. Five refuse an AI search
crawler: besides those, apitools.dev refuses PerplexityBot and every user-triggered fetcher,
code.haverbeke.berlin refuses OAI-SearchBot and PerplexityBot and also disallows them in
robots.txt, and vite-php.nititech.de rate-limits them with a 429. None of these shows in
robots.txt, which is why the audit fetches as each bot instead of parsing a file.

What did not show up

The Next.js streaming-metadata failure, where a page's title and canonical reach Googlebot
and arrive later in the body for every other agent, appeared on 0 of the 84 Next.js sites.
It appeared on two sites built with other things (an HTMX site and a Nuxt site). The failure
is real and reproducible; it is not common on the sites that ship the framework's
documentation.

Thirty-two sites, developer.mozilla.org, solidjs.com and mantine.dev among them, stamp every
sitemap URL with the same date, which is the build time and not the content's. Google uses
lastmod only while it proves accurate and ignores it once it does not, so those sitemaps
carry no freshness signal at all.

What it changed here

The run measured the tool as much as the sites. Across the 367 reports the audit printed 56
codes per site on average, and 30 of those codes appeared on six in ten of the best-built
documentation sites on the web. A notice every site gets carries no information, so 0.0.55
removed 25 of them (the Google tag manager and analytics, Bing's webmaster tools, and the
brand, entity, experience, engagement, snippet and dwell-time heuristics that had no evidence
behind them), stopped reporting a missing breadcrumb on a homepage, and stopped asking for
IndexNow on a site with no sitemap. The next weekly run is the first one taken with the
shorter list; the numbers above are from the run that caused it.

Check your own

npx @rankcli/cli audit --url https://your-site.com -o llm 

prints the same findings for one site, by code, with the fix. The corpus's scripts, list and
every weekly result are public; a site can ask to be left out on the repository.

Not in this piece, on purpose: any claim that a score predicts citations. No published score
does, and the dataset is a measure of hygiene, not of outcomes.