AI crawlers explained: the tokens, the robots.txt, and the curl that proves it

The AI crawlers that matter in 2026 answer to eleven user-agent tokens, and they split into three jobs. Training crawlers (GPTBot, ClaudeBot) teach models that you exist. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) build the live indexes that produce linked citations. On-demand fetchers (ChatGPT-User, Claude-User, Perplexity-User) grab a single page the moment a user asks for it. Allow all eleven if you want to be cited — then prove they get through, because on the SaaS sites I scan, the block is almost never in robots.txt anymore.

An AI crawler is a user-agent run by an AI company that requests your pages much like Googlebot does, except it feeds a language model or an answer engine instead of a classic ten-blue-links index.

Every token, what it does, and what blocking it costs

Read the last column before you touch your config. Blocking a training bot costs you what future models know; blocking a search bot costs you citations now; blocking a fetcher fails in front of a live prospect. Three of these tokens aren't what they look like — footnotes below the table.

Table 1 — The eleven AI crawler tokens worth knowing in 2026, by job and by cost of blocking.

The footnotes that actually matter

Google-Extended never appears in your logs. It isn't a bot. It's a robots.txt token that Google's regular crawlers read to decide whether your content may train Gemini and ground its answers. Googlebot keeps crawling either way — and AI Overviews is a Search surface, governed by Googlebot, not by this token. Block Google-Extended and you still rank. Block Googlebot and you're gone from everything at once.

Bingbot is the ChatGPT dependency nobody audits. ChatGPT's live search runs on Bing's index. Seer Interactive measured the overlap in February 2025: 87% of SearchGPT's citations matched Bing's top organic results, against 56% for Google's. Whatever your OpenAI rules say, if Bingbot can't read you, ChatGPT mostly can't cite you.

ChatGPT-User is not a click-through fetcher. It fires mid-conversation, when a user asks ChatGPT to open or read a specific page — "pull up their docs and check the rate limits." Citation eligibility in ChatGPT Search belongs to OAI-SearchBot, full stop. A lot of published advice swaps those two, which is how sites end up blocking exactly the wrong bot with complete confidence. And the Perplexity cousin plays by different rules again: Perplexity's own docs say Perplexity-User generally ignores robots.txt, on the logic that a human asked for the page.

The copy-paste robots.txt for getting cited

The cautious default — repeated in most "protect your content" posts — is to block the training bots and grudgingly allow the rest. For a publisher guarding a paywall, fine. For a SaaS that wants to be recommended inside AI answers, backwards: you can't be the cited source for a model that was never allowed to learn you exist. On a site whose entire job is getting cited — this one — I allow everything.

A plain User-agent: * with Allow: / already permits every token in the table; robots.txt only needs rules to restrict. I list the AI tokens explicitly anyway. It documents intent, and it gives you one obvious switch per bot if you ever change your mind.

# quillcite.com/robots.txt — allow everything that can cite us.
# "User-agent: * / Allow: /" already permits these; the explicit
# block below documents intent and makes audits trivial.

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Googlebot
User-agent: Google-Extended
User-agent: Bingbot
Allow: /

User-agent: *
Allow: /

Sitemap: https://quillcite.com/sitemap.xml

One trap: robots.txt groups don't merge. A bot that matches a named group obeys that group alone and ignores User-agent: * — so if your * section disallows /admin or /drafts, repeat those lines inside the AI group, because the named bots won't inherit them. The reverse trap bites harder: any bot you never listed falls through to *, where a stray Disallow: / from someone's 2023 scraper panic still blocks every AI crawler that shipped since.

Verify, don't assume

Everything above this line is permission. None of it is access. The real 2026 story is that blocking moved down the stack — out of robots.txt, into the CDN and the WAF, where nobody who writes content ever looks.

Cloudflare made it policy: from September 15, 2026, Training and Agent crawlers are blocked by default on pages that carry ads, for new domains and all Free-tier customers (announced July 1, 2026). Search crawlers stay allowed — but a multi-purpose bot gets caught wherever Training is blocked, and Cloudflare's own examples of multi-purpose crawlers include Googlebot and BingBot. Read that again next to the 87% figure above. Perplexity, meanwhile, now publishes step-by-step WAF-whitelisting guides for Cloudflare and AWS. Vendors don't write docs like that for problems that don't exist.

In July 2026 I ran the exact check below across 72 SaaS domains. writer.com — an AI company — allows every one of these bots in robots.txt while its WAF returns 403 to six of them. Only OAI-SearchBot gets through. Someone whitelisted one bot and called it done. beehiiv.com serves 429 to ChatGPT-User, which means every reader who asks ChatGPT to open a beehiiv newsletter watches the fetch fail.

A clean robots.txt is permission, not access. The only proof that an AI bot can read your site is fetching the site as that bot.

— Ruslan Kutumbaev

The check is one line:

curl -sSL -A "GPTBot" -o /dev/null -w "%{http_code}\n" https://yourdomain.com/

200 means readable. 403 is a WAF block, 429 is rate-limiting, 503 is usually a bot challenge — and every one of them means that crawler can't read you, whatever robots.txt says. Loop it across the fleet:

for UA in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot \
          Claude-User PerplexityBot Perplexity-User Bingbot; do
  printf '%-17s ' "$UA"
  curl -sSL -A "$UA" -o /dev/null -w '%{http_code}\n' "https://yourdomain.com/"
done

Healthy output is a column of 200s. Here is writer.com from my July scan, seven bots tested:

GPTBot            403
OAI-SearchBot     200   <- the one bot on the whitelist
ChatGPT-User      403
ClaudeBot         403
Claude-SearchBot  403
PerplexityBot     403
Perplexity-User   403

Two honest caveats. Run the loop once with no -A flag as a control — if bare curl also gets a 403, your WAF blocks curl itself, and your access logs become the ground truth: grep them for the tokens above and count 200s against 403s. And a spoofed user-agent catches user-agent rules, which is what my scan was catching; a WAF that verifies crawlers by IP range can treat the real bot differently than it treats your laptop. If you'd rather not open a terminal, the free access checker runs this same comparison from your browser, control request included, in about ten seconds.

One thing the status code won't show: whether the bot got your content. Most AI crawlers don't execute JavaScript, so if your pricing table only exists after React hydrates, a 200 delivers an empty shell. Drop the -o /dev/null and grep the raw HTML for the sentence you want cited. If it isn't there, neither are you.

Access is the first gate, not the win

An open bot fleet makes you readable. Cited takes more — extractable answers, schema that matches the page, third-party proof — the rest of the GEO guide for technical SaaS. And if the loop comes back all-200 while ChatGPT keeps recommending your competitor, the problem sits upstream: why your SaaS is invisible in ChatGPT covers the five causes I keep finding.

Run the curl first, though. Access is gate one of six in my GEO audit method because it's the cheapest check in the discipline and the one that fails most often. Two minutes, nine status codes, and you know whether anything else you fix can matter.

Sources and further reading

• OpenAI — overview of OpenAI's bots and user-agents
• Anthropic — Anthropic's crawlers and how site owners control them
• Perplexity — Perplexity's crawlers, IP ranges, and WAF guidance
• Google — Google crawlers and the Google-Extended token
• Cloudflare — new AI-crawler defaults from September 15, 2026 (published July 1, 2026)
• Seer Interactive — 87% of SearchGPT citations match Bing's top results (February 2025)
• Microsoft — Bing Webmaster Tools