Open your product page in Chrome and it looks great: pricing table, feature list, customer quotes. Now ask a less flattering question: what does an AI crawler see when it asks for the same URL?

For a lot of sites the honest answer is "much less than you think." Here's why, and a small script to check your own pages in a minute.

Most AI crawlers don't run your JavaScript

Googlebot renders pages: it fetches the HTML, then runs the JavaScript in a headless browser, then indexes the result. Most of the newer AI crawlers don't do that second step.

Vercel looked at real crawler traffic across its network in late 2024 and found that the major AI crawlers, including OpenAI's and Anthropic's, fetched JavaScript files but didn't execute them. Whatever your page shows only after scripts run (a pricing table loaded from an API, reviews injected by a widget, tabs rendered client-side) simply isn't in what they read. Google's own AI features are the exception, because they lean on Google's rendering pipeline.

Things may change, and crawlers don't publish their rendering behaviour in detail. But "the raw HTML is what they get" is the safe assumption for planning.

That matters for AI answers because the engines quote what they can read. If your pricing only exists after hydration, an engine answering "how much does X cost?" has to get it from someone else's page.

The check: fetch it like a bot, strip the scripts

The script below asks for a URL with the published user agents of GPTBot (OpenAI's training crawler), OAI-SearchBot (the one behind ChatGPT search results) and PerplexityBot, throws away <script>, <style>, <noscript> and <template> blocks, and reports what's left:

  • the HTTP status (some firewalls quietly return 403 to bot user agents),
  • how many words of real text remain,
  • the title, meta robots and canonical,
  • whether the phrases you care about are actually in there.

No dependencies, Python 3.8+.

#!/usr/bin/env python3 """What does an AI crawler actually receive from your page? Usage: python3 what_bots_see.py https://example.com/pricing "$29/mo" "free plan" """ import re import sys import urllib.error import urllib.request from html import unescape # User agents as published by OpenAI and Perplexity (checked 14 Nov 2026). AGENTS = { "GPTBot": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot", "OAI-SearchBot": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot", "PerplexityBot": "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)", } def fetch(url, agent): req = urllib.request.Request(url, headers={"User-Agent": agent}) try: with urllib.request.urlopen(req, timeout=20) as r: return r.status, r.read().decode("utf-8", "replace") except urllib.error.HTTPError as e: return e.code, e.read().decode("utf-8", "replace") def visible_text(html): html = re.sub(r"(?is)<(script|style|noscript|template)\b.*?</\1>", " ", html) text = unescape(re.sub(r"(?s)<[^>]+>", " ", html)) return re.sub(r"\s+", " ", text).strip() def find(pattern, html): m = re.search(pattern, html, re.I | re.S) return unescape(m.group(1).strip()) if m else "-" def main(): if len(sys.argv) < 2: sys.exit(__doc__) url, phrases = sys.argv[1], sys.argv[2:] for name, agent in AGENTS.items(): status, html = fetch(url, agent) text = visible_text(html) print(f"\n== {name}: HTTP {status}, {len(html):,} bytes of HTML, {len(text.split()):,} words of text") print("title: ", find(r"<title[^>]*>(.*?)</title>", html)) print("meta robots:", find(r"<meta[^>]+name=[\"']robots[\"'][^>]+content=[\"']([^\"']*)", html)) print("canonical: ", find(r"<link[^>]+rel=[\"']canonical[\"'][^>]+href=[\"']([^\"']*)", html)) for p in phrases: print(f" {'FOUND ' if p.lower() in text.lower() else 'MISSING'} {p!r}") if __name__ == "__main__": main()

Reading the output

Run it with two or three phrases you can see on the page in your browser: a price, a product name, a sentence from your main claim.

python3 what_bots_see.py https://yoursite.com/pricing "$29/mo" "Free plan"

What to look for:

  • HTTP 403 or 503 for the bot but 200 in your browser. A WAF or bot-protection rule is blocking the crawler. Decide whether that's what you want; often nobody decided.
  • A few dozen words of text on a page that looks long in Chrome. The content is being rendered client-side. That's the classic single-page-app result.
  • MISSING on a phrase you can see on screen. Either it's rendered by JavaScript, or it's inside an image, a canvas or an iframe. Same outcome for the crawler.
  • noindex in meta robots, or a canonical pointing somewhere else. Easy to ship by accident from a staging config.

One honest caveat: a MISSING phrase isn't always a JavaScript problem. If the page says "$29/mo" and you search for "per month", the script will say MISSING because those words really aren't there. Search for the exact text you see.

Fixing what you find

You don't need to rebuild the site. In rough order of effort:

  1. Server-render or pre-render the pages that answer buyer questions: pricing, comparison pages, "what is X", integrations. Most frameworks (Next.js, Nuxt, Astro, SvelteKit) can do this per route.
  2. Put the key facts in the HTML even if the fancy version loads later. A plain table of plans and prices that the script then enhances is fine.
  3. Don't hide answers in tabs or accordions that only exist after a click handler runs. If the text is in the HTML and just collapsed with CSS, crawlers can read it.
  4. Check bot rules on your CDN or WAF for the crawlers you actually want, and leave the ones you don't.

Then run the script again and watch the word count go up.

Doing this for a whole site

The script checks one URL at a time on purpose; it's for spot checks. If you'd rather not loop it over your sitemap yourself, the AI readiness audit in Presence Scout runs a broader version of this check (robots.txt rules, what a crawler actually receives, real text, schema, canonicals) on up to 50 pages of your site, next to how often ChatGPT, Gemini and Perplexity name your brand. Full disclosure: I work on it. Here's a two-minute walkthrough:

Whatever you use, the habit is the useful part: every time you ship a new page that should answer a buyer's question, fetch it like a bot once.