A Practical Guide to robots.txt Rules and Common Mistakes

A robots.txt file tells cooperating crawlers which paths on a site they may request. It is useful for managing crawling, but it is neither an access-control mechanism nor a guarantee that a URL will be removed from search results.

The examples below use the reserved domain www.example.com and fictional paths. Adapt them to your own site after checking the current rules and the behavior of each crawler you care about.

Crawling, indexing, and privacy are different problems

Crawling is a robot fetching a resource. Indexing is the search engine's later decision to store and potentially show information about it. Allowing a URL in robots.txt does not guarantee indexing or rankings. Blocking its crawl does not necessarily remove a URL already known through links or other sources.

A noindex instruction belongs in a supported page-level mechanism, such as a robots meta tag or an HTTP response header, not as a substitute line in robots.txt. To observe that instruction, a crawler generally needs to be able to request the page and read the relevant response. If robots.txt blocks the request, the search engine may never see the page's noindex.

Robots.txt is public and can be ignored by non-cooperating clients. Protect private documents and account areas with authentication and server-side authorization. Don't use a crawler rule as a password.

Location and a minimal example

Publish the file at the root path for the appropriate site origin, such as:

https://www.example.com/robots.txt

Check the exact scheme, host, and, when relevant, port for the URLs you are investigating. A subdomain or the version without www needs its own check; don't assume the file on one host automatically governs another.

Use a plain UTF-8 text file. This illustrative file contains a broad restriction and a more specific allowed path:

User-agent: * Disallow: /demo-search/ Allow: /demo-search/help/ Sitemap: https://www.example.com/sitemap.xml

Here the general group tells applicable crawlers not to fetch paths under /demo-search/, while listing an exception for /demo-search/help/. Test the exact paths and crawler before deploying. The Sitemap line must point to a real sitemap for your site; announcing it does not override crawl restrictions or guarantee indexing.

Understand groups and path matching

User-agent groups

A User-agent line identifies the crawler a group is meant for. The asterisk identifies a general group. Add a crawler-specific group only when there is a clear operational reason, because additional groups make rules harder to audit.

For Google, don't assume the general group's rules are automatically appended to a matching, more specific group. Check the applicable group and test the same URLs for each crawler. Other search engines may have their own group selection details, so consult their documentation.

Disallow and Allow

Disallow describes a restricted path; Allow can express an allowed path, including a more specific exception. When rules overlap, the result depends on matching and the crawler's implementation. Test a complete URL with the relevant crawler rather than assuming that the last line wins.

Pay attention to the trailing slash. A rule beginning /catalog can also match another path that starts with the same characters. /catalog/ targets the path with that slash and the paths below it; test /catalog without the slash separately. Path case matters in matching as well.

Treat wildcard patterns and query strings as changes requiring a sample of real URLs. A broad rule designed to limit search pages could accidentally affect useful parameterized pages or resources. Document which URLs should remain crawlable before applying it.

Sitemap and comments

A Sitemap entry points to an absolute sitemap URL. Verify the map is accessible and contains the appropriate current URLs. Listing it is a discovery aid, not a promise that every URL will be crawled or indexed.

Comments begin with # and can explain why a rule exists. The file is public: never include secrets, personal data, or sensitive internal instructions in comments.

A safe workflow for changing rules

Record the existing file. Save its contents, the date, and how it is generated or served. A CMS or server configuration may create it dynamically.
Build a representative URL set. Include the homepage, important product or service pages, restricted public test paths, required CSS and JavaScript, and parameterized pages in use.
Write expected results. For each complete URL and relevant crawler, note whether crawling should be allowed or disallowed and why.
Prepare the smallest necessary change. Preserve the existing rules until their purpose is understood; don't replace the whole file with an unrelated template.
Test before and after deployment. Check special groups and adjacent paths, then fetch the public file and compare the result with the approved text.
Monitor. Search systems may not use the new file instantly. Record the change and look at crawler reports and server logs rather than promising a specific reaction time.

You can inspect a response using a request such as:

curl -i https://www.example.com/robots.txt

Replace the domain with your own. Inspect the final HTTP response, headers, and body; investigate redirects if present. A normal configuration commonly returns the expected text with HTTP 200. If the response is a login form, an error page, or the homepage HTML, correct the serving configuration.

Test similar URLs separately

Imagine that /demo-search/ is a public demonstration section whose crawl the owner wants to restrict, while /demo-search/help/ should remain available. The example file gives a broad disallow and a more specific allow. Test both full URLs with the crawler's available tools, along with their relevant parameterized variants and required page assets.

Also test /demo-search without the trailing slash. It is a distinct path, so don't infer the result solely from a test of /demo-search/. If the content is actually confidential, set access controls first; this example is about crawling, not privacy.

Common mistakes

Leaving a staging restriction on production. Disallow: / in an applicable group restricts the whole path space for that crawler. Verify the domain and the reason for the rule before changing it.
Confusing crawl control with deindexing. A blocked crawler may not see a page's noindex. Choose the indexing or access-control mechanism appropriate to the goal.
Blocking assets needed to understand a page. Check important CSS, JavaScript, and images; don't unblock every technical path without understanding its purpose.
Copying unsupported directives. Search engines can differ. Google does not support using Noindex in robots.txt as a way to exclude a URL from its index.
Testing only one crawler or one spelling of a URL. Include specific user-agent groups, trailing slashes, path case, and representative query strings in the test set.
Assuming a sitemap is an indexing command. It helps communicate URLs but does not override access rules or guarantee results.