We told Google not to index 10,538 of our 24,667 pages. Two batches of those were our mistake.

Numbers below come from one directory site, anclum.com, in the 24 hours to 2026-11-11. Every figure has a command next to it at the end.

Our sitemap has 9,842 URLs. Our site has 24,667 HTML pages. That gap is not an accident:

all pages 24,667 sitemap URLs 9,842 pages with noindex 10,538 

43% of our own pages carry a noindex tag. We put it there. This post is about the part of that number that was wrong.

Why a directory would exclude half of itself

A directory grows by generating pages from data. That produces three kinds of pages you should not ask Google to index:

1. Thin pages. An entry with a name and a one-line description is not a page worth ranking. If the description is under ~120 characters and there is no README, we mark it thin.
2. Duplicate copies. The same entry reachable under two slugs. One canonical, one excluded.
3. Untranslated pages. We publish an English tree and a Chinese tree. When the Chinese page is really the English body with a Chinese shell, marking it noindex is honest: it is not Chinese content.

That policy accounts for most of the 10,538. Of the excluded pages, 9,744 are the zh tree and 794 are the English tree (792 of those are entry detail pages).

The policy is not the interesting part. The interesting part is that we twice excluded pages that did not belong in that list — and both times, the bug was the same shape.

Mistake 1: we read a different field than the page renders

Our thin rule originally looked at one column, one_line, plus whether a README body existed. The page template, however, also renders official_desc and several other fields.

So the rule was asking "is this row short?" while the page was answering "here is a paragraph."

When we changed the rule to measure the text a reader would actually see in <main>, 307 pages stopped being thin. Those pages never changed. Their data never changed. Only the ruler did.

Mistake 2: the data was there, the template was not showing it

The second batch is the one that stings, because nothing was mis-measured. The pages really were thin — on the rendered page.

But for 58 English pages, the underlying record held more description text than the template was displaying: a continuation of official_desc that our layout had never rendered. We had the words; we only published the first part.

We rendered that continuation for those 58 pages, behind an explicit allowlist, and the result was exactly the expected pair:

sitemap 9,784 → 9,842 (+58) noindex count 10,596 → 10,538 (−58) 

Those pages carried 43–110 characters of their own description before the change. After it, a sample page renders 1,499 characters of body text. Same database row. Same entry. One template change.

The shape of both mistakes

Both are the same error, and it is not a coding error. It is a question error:

We were judging the page by a proxy — a column, a flag, a precomputed list — instead of by the thing the judgement was actually about.

The judgement we wanted to make was "would a reader find this page useful?" The judgements we were actually making were "is this column long?" and "did the template print this field?"

Those are cheaper to compute. That is exactly why they get used. And they are wrong in a way that never throws an exception: the site builds, the pages render, the sitemap validates, and 365 pages that should be indexed quietly are not.

What we do now

Three rules, each earned by a specific failure this week:

1. The input to a judgement must be what the reader sees. If the rule is about page content, feed it the rendered content.
2. A criterion that is red in a legal state is not testing a defect. Our first internal-link criterion flagged links to any noindex page — but we deliberately noindex static pages and translate gaps, and those are legitimately linked. The criterion now asks a narrower, answerable question: links to pages marked noindex because they are thin.
3. Report the subtraction, both sides. Adding 58 URLs to the sitemap while removing 58 from the noindex set is a paired change. If only one side moves, something else is wrong and you should stop.

The honest limit

We have not audited all 10,538. We audited the English detail pages, which is 792 of them, and found 58. The zh tree (9,744 pages) we deliberately have not touched — excluding untranslated pages is a policy, not a bug.

There is also one page in our thin list that returns nothing at all — the file is not on disk. We do not yet know whether its slug changed or the entry was removed. That is on the list, not swept under it.

If you run a generated site, the useful exercise is not "how many pages do I have." It is: take every rule that hides a page from search, and ask what its input actually is. If the answer is a column name rather than rendered text, you have a batch of pages you have not seen yet.

Numbers and where they came from