When a buyer asks ChatGPT, Claude, Gemini or Google's AI Overviews which tool to use, the assistant can only describe your product from what it was able to fetch, or from what other sites say about you. Plenty of teams spend weeks on "AI SEO" content and never check the basics underneath. Is the site blocked, does the content only appear after JavaScript runs, and is there one clear page that explains what the product does?

This is a 15-minute check you can run on your own site with nothing more than a browser. None of it needs a paid tool.

1. Read your robots.txt like an AI crawler would

Open https://yourdomain.com/robots.txt and look for rules aimed at AI user agents. The ones you will most often see are:

  • OpenAI: GPTBot (model training), OAI-SearchBot (ChatGPT search results) and ChatGPT-User (pages fetched when a user asks).
  • Anthropic: ClaudeBot, plus the search and user-fetch agents Anthropic documents.
  • Perplexity: PerplexityBot.
  • Google: Google-Extended is a control token, not a separate crawler. AI Overviews and AI Mode are built on regular Google Search crawling, so a Disallow for Googlebot hurts you everywhere.
  • Microsoft: Copilot answers lean on the Bing index, so Bingbot access matters.

Vendors rename and add agents, so check each one's current documentation before you edit anything.

Three patterns cause most of the accidental blocks:

  1. A blanket User-agent: * / Disallow: / left over from a staging site.
  2. A security plugin or CDN rule that added AI bots to a block list, sometimes switched on by default. In this case robots.txt looks fine, but the firewall returns 403.
  3. Blocking the search agent when you meant to block training. Some teams want to opt out of training but still be cited in answers. Those are different user agents with different rules.

Decide on purpose. If you want to be named in AI answers, the agents that fetch pages for answers need access to your public marketing pages.

2. Test what a non-browser client receives

Robots rules are only a request. What matters is what the server actually returns. From a terminal:

curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://yourdomain.com/

Then run it with a normal browser user agent and compare the two. A 403, a 429 or a challenge page for the bot means a firewall rule is blocking it, whatever robots.txt says.

Then check the page body itself. Run curl -s https://yourdomain.com/pricing/ | grep -i "per month", or grep for any phrase you know is on the page. If the text only shows up after client-side JavaScript runs, many fetchers will see an empty shell. Your pricing, feature list and "what is it" paragraph should be in the HTML the server returns.

3. Make sure there is a sitemap and that it is current

Robots.txt should point to it (Sitemap: https://yourdomain.com/sitemap.xml). Open the sitemap and spot-check that:

  • the key pages are there (home, product, pricing, docs or guides),
  • URLs return 200, not redirects or 404s,
  • staging, tag or search-result URLs are not padding it out.

4. Give machines a short, honest summary: llms.txt

llms.txt is a proposed plain-markdown file at your root. It gives a one-paragraph description of the site and links to the pages that matter most. It is not a ranking switch and not every assistant reads it, but it costs ten minutes and forces you to write the clearest possible description of your product. That description is useful everywhere else too.

Keep it factual. Say what the product is, who it is for and what it costs, then link your best explainer pages. Don't stuff it with keywords.

5. Check that your key facts are extractable

Assistants quote sentences, not layouts. Open your homepage and pricing page and ask:

  • Is there one plain sentence that says what the product is? ("X is a ... for ...")
  • Are prices written as text, not only inside images or toggles?
  • Do headings match the questions buyers ask ("How much does X cost?", "Does X integrate with Y?")?
  • Is there structured data (Organization, Product or SoftwareApplication, FAQPage where it fits) that matches the visible text?
  • Do canonical tags point at the right URL, so duplicate versions don't split signals?

If an assistant can't find a fact in one or two clean sentences, it will either skip you or fill the gap with whatever a third-party page says.

6. Write down what you found, then re-check after changes

Make a short table with five columns: check, status, page, fix and owner. Most sites end up with two or three real issues, usually a firewall rule, a JavaScript-only pricing table or a missing sitemap entry. Fix those first. Wording tweaks can wait.

Doing it faster

If you'd rather not run each step by hand, Suparank has a free agent readiness checker. It needs no sign-up and scores a site from 0 to 100 against about 20 AI-agent standards, covering robots rules, sitemaps and the well-known files agents read. For a full crawl, the GEO and technical audit in Suparank runs 63 checks in 11 categories, including AI crawler access and llms.txt, and ranks the fixes by impact.

Either way, fix access first. Content work only pays off once assistants can actually read your pages.