A link audit can return a reassuring list of HTTP 200 responses and still miss an error page. It can also report a working link as broken because the destination blocks automated requests.

A useful audit separates three questions: did the request succeed, where did it end up, and does the destination contain the page the visitor expected? Here is a practical workflow for website maintenance, content reviews and migration checks.

1. Separate the findings before fixing them

Finding What it tells you What to check next
HTTP 404 or 410 The requested resource was not found or is gone Remove the link or choose a relevant replacement
HTTP 5xx The server failed to fulfil this request Retry later and investigate if it persists
DNS, TLS or timeout failure The checker could not complete the connection Verify the domain, certificate or connection
HTTP 401, 403 or 429 Authentication, refusal or rate limiting affected the request Mark it unverified and investigate access
Redirect chain Reaching the destination took multiple redirects Inspect the entire chain and update links where appropriate
HTTP 200 with error-page content The status alone does not establish that the page exists Review it as a possible soft 404

These are observations from a check, not promises about future availability. The HTTP specification defines the response codes. For a 429 response, follow the server's rate-limit instructions, including Retry-After when provided; the status does not prove that the resource has disappeared. RFC 6585.

2. Keep the source page, not just the dead URL

Start with the homepage and follow internal links. Add sitemap URLs if you want to include pages that are not reachable through the navigation. A sitemap is an extra discovery source, not proof that every page has been found.

For each destination, keep:

  • the URL as linked and its resolved absolute URL;
  • the source page and visible link text;
  • whether the reference is a link, image, script or stylesheet;
  • the response code, final URL, redirect hops and time checked.

Deduplicate requests to the same destination, but retain its source pages. If an outdated URL occurs throughout a navigation template, that is a different repair from one old reference in a blog post.

For a large site, set a page cap and record whether the crawl reached it. Respect robots.txt and the site's request limits. A checker that reads server-delivered HTML will not discover links that appear only after JavaScript executes.

3. Inspect redirects with a real GET request

For one suspicious URL, curl is enough to inspect the HTTP chain. This Bash example uses an illustrative URL; replace it with a page you are checking.

curl --location \ --max-redirs 10 \ --max-time 30 \ --dump-header response-headers.txt \ --output response-body.html \ --write-out 'status=%{http_code}\nredirects=%{num_redirects}\nfinal=%{url_effective}\n' \ 'https://example.com/old-page' 

The headers file contains the responses along the redirect path, including their Location headers. The body file contains the final response. The redirect and time limits are settings for this example. Without --fail, curl exits with 0 even on a 404, so read the printed status rather than the exit code. If curl exits with an error (for example code 6 for an unknown domain, which prints status=000), keep that failure in your notes instead of treating the printed status as a successful check. curl option reference.

A chain such as old-page → previous-page → current-page is an illustrative case where an internal link could often point directly to current-page. Confirm that the destination is the intended replacement before changing it. Cross-domain redirects, temporary redirects and campaign tracking may be intentional.

Do not remove existing server redirects merely because you updated an internal link: external links and bookmarks can still use the old address. Also check for redirect loops or excessive hops. A single failed run cannot tell you whether the condition is permanent.

A HEAD request can be useful for a lightweight first pass, but it does not return the page body. If a server rejects HEAD or gives a suspicious result, verify with GET. Do not repeatedly retry a rate-limited or access-denied endpoint.

4. Check soft 404s as a separate content problem

Google describes a soft 404 as an error-like or empty page that returns a success status, commonly HTTP 200. Its Page Indexing report can flag URLs this way. A status-code-only checker cannot reliably identify the same condition. Google's soft-404 guidance.

For a suspected page:

  1. Open the final URL and inspect its main content. Check the rendered page if JavaScript supplies the content.
  2. Compare it with a known missing URL on your own site. Repeated error headings or an identical error template can provide a clue.
  3. Check whether an old URL redirects to an unrelated homepage or generic search page.
  4. For a site you manage, use Search Console's Page Indexing report and URL Inspection to see what Google fetched or rendered.

Treat automatic content matching as a candidate detector. A legitimate article about “page not found” should not become a confirmed soft 404 just because it contains those words. Language, templates and rendering failures can all affect a heuristic.

If the resource has been removed without a relevant replacement, return a proper 404 or 410. If it has moved, redirect to the relevant replacement. If the content should exist, repair the page or its rendering. Google documents these different fixes in the same guidance.

5. Run the crawl without maintaining a crawler

I built Broken Link Checker on Apify to collect HTTP and connection failures, redirects and source-page references.

A starting configuration is one website and a modest maxPagesPerSite (the default is 100). The defaults useSitemap and respectRobotsTxt are on, which suits a first run. Keep checkResources on if missing images, scripts and stylesheets are part of the audit. Leave onlyBroken off to keep redirect findings, and reportUnverifiable on to retain results that need manual inspection.

Review the output by result:

  • broken: investigate the failure and choose a repair;
  • redirect: inspect the destination and chain;
  • unverifiable: check access or retry conditions;
  • warning: review the reported issue, such as an HTTP reference on an HTTPS page.

The output includes source pages and link text. Source-page examples are capped at ten per problem link; the occurrence count helps show whether there are more references. Check pageLimitReached in the site summary before treating the crawl as complete.

Current limitation: this Actor checks response status and redirects; it does not analyse page content to detect soft 404s. It also does not render JavaScript. Use the separate content and Search Console checks above for that part of the audit.

Save a tested configuration as a task if you want to repeat it, then use an Apify schedule. Review current Store pricing and run limits before enabling recurring runs.

(Disclosure: I built it. The curl and manual checks above work independently of the Actor.)

6. Recheck the repair

Update the link or template, then revisit the source page and follow the destination again. Confirm both the HTTP behaviour and the content. Keep uncertain results separate from confirmed missing pages so that a blocked request does not lead to deleting a useful reference.

What produces more work on your site: vanished external sources, old internal URLs or pages that look missing while returning HTTP 200?