I crawled all 530 of our own pages: no orphans, but 19 have exactly one way in

Our sitemap has 530 URLs. A sitemap says a page exists; it says nothing about whether anything on the site points at it. So I fetched all 530, pulled every <a href> out of the rendered HTML, and built the graph.

The method, including the part that bites

stripped = re.sub(r"(?is)<script.*?</script>|<style.*?</style>", "", html) links = re.findall(r'<a\b[^>]*?href="([^"]+)"', stripped) 

The re.sub is not cosmetic. This is a React Server Components app, and the flight payload at the bottom of the document repeats every string the page rendered, including the hrefs. Count links in the raw HTML and you count a good share of them twice. Strip the script tags first, then count. (Also normalise trailing slashes and resolve relative hrefs, or /tests and /tests/ become two different nodes in your graph.)

All 530 answered 200. Page sizes ran from 67,808 to 737,954 bytes.

The floor: 34 links every page already has

35 destinations appear on at least 95% of pages. Two of them, /login and /signup, appear on all 530; the other 33 appear on 529. That is the header and the footer, and it means no page on the site has an out-degree below 34.

Six pages sit exactly on that floor, linking to nothing except what every page links to:

/pricing /comparison /about /help /terms /cookies 

At the other end:

Mean out-degree across the site is 43.3. Four indexes and one quiz-style article carry most of the real linking.

In-degree: nothing is orphaned

Zero of the 530 pages have an in-degree of zero. That is the number I was actually checking, because an orphan is a page we paid to write and then told nobody about except a crawler reading XML.

The distribution underneath that is more honest:

The 35 chrome destinations are in that last bucket by definition, sitting at 529 or 530.

The 19

All 19 pages with exactly one inbound link are employer guides, and in all 19 cases the one page linking to them is the /employers index:

/employers/allstate /employers/arcadis /employers/bp /employers/bt-group /employers/home-depot /employers/ifs /employers/medela /employers/metal-supermarkets /employers/mintek /employers/ors /employers/panera-bread /employers/rio-tinto /employers/sky /employers/teach-first /employers/tesco /employers/tlc-care /employers/tui /employers/vena-solutions /employers/view-group 

That is not a bug, it is a shape. We have 180 employer pages. The footer hand-picks four of them (Amazon, Barclays, HSBC, PwC), which is why those four have 529 inbound links and everything else in the cluster has between 1 and 20. 41 of the 180 have two or fewer.

The same shape shows up next door: 25 of the 56 /cheating/<provider> pages have exactly two inbound links, which are their index and the matching provider hub. And two of the 50 per-test pages have two, because the only pages that link to them are their own hub and each other:

/games/cappfinity/numerical-and-verbal-reasoning <- /games/cappfinity, /games/cappfinity/situational-strengths-test /games/cappfinity/situational-strengths-test <- /games/cappfinity, /games/cappfinity/numerical-and-verbal-reasoning 

Where an employer page does have contextual inbound links, they come from the provider hub for the test that employer uses, which is the association being asserted in both directions. /employers/lner is linked from /games/train-driver; /employers/ubs from /games/aon. The cluster is cross-linked exactly where we had evidence to cross-link it, and flat where we did not.

Two other things the graph coughed up

23 internal link targets are not in the sitemap at all. Twenty of them are auth-gated practice routes (a separate problem, which I wrote about yesterday), /login and /signup are deliberate, and the twenty-third is this one:

/dashboard/settings <- /privacy 

One public page links into the signed-in interior of the app, and it is the privacy policy pointing at the controls it describes. That is the correct page to be the exception.

The live site also links out to 324 distinct external URLs across 140 hosts. That is the next audit, and it has a nastier failure mode than this one.

See it

Pick any of the 19 from the list and look for yourself. Tesco's guide is one: open it, and the only internal links are the chrome plus its own content links outward. Then open the employer index, hit Ctrl+F and search tesco. That single anchor is the whole inbound side of that page.

The three-line version, if you want the numbers for your own site:

curl -s https://cogniprep.app/sitemap.xml | grep -o '<loc>[^<]*' | sed 's/<loc>//' | wc -l 

then fetch each one, strip the script tags, count the anchors, and sort by in-degree. The one query worth running first is in-degree == 0. The one worth running second is in-degree == 1, because that is where a page is one editorial decision away from being invisible.