Why Google Has Crawled Your Pages but Still Has Not Indexed Them

You publish a new page, submit your sitemap, and wait for Google to discover it.

A few days later, you open Google Search Console and find this status:

Crawled - currently not indexed

The page is live. It returns a successful HTTP response. Googlebot has already visited it. Yet the URL is still missing from Google's search index.

For developers and technical SEO professionals, this can be confusing.

If Google has already crawled the page, why is it not indexed?

The answer is that crawling and indexing are two different processes. A successful crawl does not guarantee that Google will include a URL in its search index.

This article explains the difference between crawling and indexing, why these statuses appear, and how developers can investigate them systematically.

1. Understanding Crawling vs. Indexing

Before debugging an indexing issue, it is important to understand how Google processes URLs.

A simplified search workflow includes:

1. Discovery
2. Crawling
3. Rendering and processing
4. Indexing
5. Serving search results

Discovery

Google learns about a URL through links, sitemaps, and other discovery mechanisms.

Crawling

Googlebot requests the page and attempts to retrieve its content.

Rendering and processing

Google processes the page, including JavaScript-rendered content where necessary.

Indexing

Google evaluates the page and may include it in its search index.

A page can successfully pass the crawling stage without being selected for indexing.

Google's documentation also makes clear that not every known URL needs to be indexed. Duplicate pages, alternate URLs, and pages that do not provide useful independent content may appropriately remain outside the index.

This means developers should investigate the reason behind a status rather than treating every non-indexed URL as an error.

2. What Does "Crawled - Currently Not Indexed" Mean?

This status means Google crawled the URL but did not include it in the index at that time.

It does not automatically mean that the website has a technical error.

Possible factors to investigate include:

• Content that provides limited independent value.
• Pages that are substantially similar to other URLs.
• Weak internal linking and unclear site structure.
• Rendering problems that affect the main content.
• Incorrect canonical signals.
• Pages that Google does not consider necessary to index.

The important point is that the status alone does not identify one universal cause.

How to investigate

Open Google Search Console and inspect the affected URL.

Review:

• Last crawl date.
• Page fetch status.
• Indexing allowed status.
• User-declared canonical.
• Google-selected canonical.
• Rendered HTML.
• Page content and internal links.

Do not immediately resubmit the URL without checking the page.

3. What Does "Discovered - Currently Not Indexed" Mean?

This status is different.

Google has discovered the URL but has not crawled it yet.

Google's documentation explains that this can happen when Google discovers a URL but postpones crawling, potentially to avoid overloading the website.

For developers, this makes crawl discovery and crawl efficiency important areas to investigate.

Common things to check

• Is the website generating too many unnecessary URLs?
• Are there large numbers of filter or parameter combinations?
• Are important pages linked from relevant pages?
• Is the XML sitemap accurate?
• Are server response times reliable?
• Are there repeated redirect chains?

A useful first step is to inspect the website's URL inventory.

For example, an e-commerce website might expose thousands of combinations through filters:

/products?color=blue /products?color=red /products?sort=price /products?sort=popular /products?color=blue&sort=price 

If many URLs provide substantially similar content, the website may be presenting a much larger crawlable URL space than necessary.

Developers should decide which URL variations deserve independent indexing and manage the rest appropriately.

4. Check the HTTP Response Before Changing SEO Settings

A page may load in a browser while its response behavior is different from what developers expect.

Start by checking the HTTP headers:

curl -I https://example.com/blog/example-page 

Look for:

HTTP/2 200 content-type: text/html 

A valid content page generally needs to return a successful response.

However, a 200 response does not guarantee indexing. It only tells you that the server successfully returned a response.

Also check the redirect destination:

curl -L -I https://example.com/blog/example-page 

The -L option follows redirects.

Investigate:

• Unexpected 301 or 302 redirects.
• Redirect chains.
• 403 responses.
• 404 responses.
• 5xx server errors.
• Inconsistent responses between requests.

A page that intermittently returns a server error may create crawling reliability problems.

5. Check for Accidental Noindex Directives

One of the most important technical checks is the robots meta tag.

For example:

<meta name="robots" content="noindex"> 

This tells search engines that support the directive not to index the page.

The page can still load normally for visitors.

Developers should inspect both the HTML and HTTP response headers because indexing directives can be delivered through either method.

For example:

X-Robots-Tag: noindex 

If a page should be indexed, remove the unintended directive.

A common development mistake is allowing staging configuration to reach production.

For example, a framework might use environment-specific settings:

const isProduction = process.env.NODE_ENV === "production"; const robotsContent = isProduction ? "index, follow" : "noindex, nofollow"; 

The exact implementation depends on the framework, but the principle is to keep staging and production indexing policies separate.

Important: Robots.txt and noindex are not interchangeable. Robots.txt controls crawling access, while noindex is an instruction to keep a page out of the index. If Google cannot crawl a blocked page, it may not see the noindex directive inside that page.

6. Verify Your Canonical Tags

Canonical problems are another important reason why a page may not appear as an independent indexed URL.

Consider these URLs:

https://example.com/blog/technical-seo https://www.example.com/blog/technical-seo https://example.com/blog/technical-seo/ 

Depending on the website configuration, these URLs may represent the same content.

A canonical tag provides a signal about the preferred URL:

<link rel="canonical" href="https://example.com/blog/technical-seo" /> 

For pages intended to be independently indexed, check whether the canonical points to the correct URL.

Also review:

• HTTP versus HTTPS.
• WWW versus non-WWW.
• Trailing slash consistency.
• Redirect destinations.
• Internal links.
• Sitemap URLs.

Do not assume Google will always follow the canonical you declare. Google may select a different canonical based on its evaluation of the available signals.

Use URL Inspection to compare the user-declared canonical with Google's selected canonical.

7. Investigate JavaScript Rendering Problems

Modern applications often use React, Vue, Angular, and other JavaScript frameworks.

These applications can create excellent user experiences, but they introduce additional considerations for search engines.

Consider a React application that loads its article content from an API:

import { useEffect, useState } from "react"; function ArticlePage() { const [article, setArticle] = useState(null); useEffect(() => { fetch("/api/article") .then((response) => response.json()) .then((data) => setArticle(data)); }, []); if (!article) { return <p>Loading...</p>; } return ( <main> <h1>{article.title}</h1> <p>{article.content}</p> </main> ); } 

The initial HTML may not contain the article's main content.
https://goodoff.co/
Google can render JavaScript, but rendering is a separate processing step and may not happen at the same time as the initial crawl.

If the application has broken API requests, blocked resources, or JavaScript errors, the rendered page may be incomplete.

How to debug

1. Open the URL Inspection tool.
2. Run a live test.
3. Examine the rendered page.
4. Compare the rendered content with the browser version.
5. Check browser console errors.
6. Inspect failed network requests.
7. Verify that important content does not depend on user interaction.

Consider server-side rendering

For content-heavy pages, server-side rendering or static generation can make important content available in the initial HTML.

For example, a framework can generate the page with its title and main content before sending the response.

This can improve reliability and reduce dependence on client-side rendering for essential content.

However, server-side rendering is not a guaranteed indexing solution. Content quality, canonical signals, crawlability, and other factors still matter.

Google's JavaScript SEO documentation explains how Google processes JavaScript pages.

8. Improve Internal Linking and URL Discovery

A sitemap can help Google discover URLs, but internal links remain an important part of website architecture.

Imagine a website publishes 100 articles, but 30 of them are not linked from any relevant category or existing article.

These pages may be difficult for users and crawlers to discover through the site's navigation.

A better structure could look like this:

Homepage | +-- Blog | | | +-- Technical SEO | | | | | +-- Article A | | +-- Article B | | | +-- Web Performance | | | +-- Article C | +-- Resources 

Use descriptive internal links that help visitors understand where a link leads.

For example:

<a href="/blog/javascript-seo"> Learn about JavaScript SEO </a> 

Avoid relying entirely on sitemaps to communicate the importance and relationships of your content.

9. Audit Your XML Sitemap

An XML sitemap should contain the canonical URLs you want search engines to discover.

A basic example:

<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" > <url> <loc>https://example.com/blog/article-one</loc> </url> <url> <loc>https://example.com/blog/article-two</loc> </url> </urlset> 

Developers should check that:

• URLs return successful responses.
• URLs use the preferred HTTPS version.
• Redirecting URLs are not included as canonical entries.
• Noindex pages are excluded.
• Duplicate URLs are consolidated.
• Important updated pages have accurate modification dates.

An XML sitemap is a discovery tool, not an indexing guarantee.

Google may discover and crawl a URL from a sitemap but still decide not to index it.

10. Check Whether the Content Deserves a Separate URL

Not every indexing problem can be fixed through code.

Consider a website with 50 pages that all explain almost the same topic.

Each page has a slightly different title, but the main content is nearly identical.

From a technical perspective, the pages may load correctly and return successful responses.

However, Google may decide that several URLs do not provide enough distinct value to deserve separate indexing.

Developers and content teams should review:

• Whether the page serves a distinct purpose.
• Whether it answers a different user need.
• Whether the content is original and useful.
• Whether it repeats information available elsewhere.
• Whether the page should be consolidated with another URL.

The solution may be improving the content, consolidating duplicate pages, or removing unnecessary URLs.

Adding more words does not automatically make a page more valuable.

11. A Practical Debugging Workflow

Here is a repeatable workflow for investigating important URLs that are not indexed.

This workflow helps separate technical problems from content and indexing decisions.

12. Useful Commands for Developers

You can use these commands as part of a basic indexing investigation.

Check response headers:

curl -I https://example.com/page 

Follow redirects:

curl -L -I https://example.com/page 

Download the HTML source:

curl -L https://example.com/page -o page.html 

Search the downloaded HTML for robots directives:

grep -iE 'robots|canonical' page.html 

Inspect the robots.txt file:

curl https://example.com/robots.txt 

These checks do not reproduce Google's complete crawling and rendering systems. They provide useful first-level evidence that developers can compare with Search Console results.

Final Thoughts

When Google crawls a page but does not index it, the correct response is not to repeatedly request indexing or make random technical changes.

First, identify the exact status. Then investigate the HTTP response, indexing directives, canonical configuration, JavaScript rendering, internal links, sitemap, and content quality.

The distinction matters:

• Discovered - currently not indexed: Google knows the URL but has not crawled it.
• Crawled - currently not indexed: Google crawled the URL but has not indexed it.
• Indexed: Google has included the URL in its index, although this does not guarantee that it will appear for every search.

A healthy website does not need every possible URL indexed. It needs its important, useful, canonical pages to be discoverable, accessible, and eligible for indexing.

For developers, the best approach is to treat indexing as part of website architecture and quality assurance rather than a task to consider only after deployment.