AI Crawlers Explained: What ClaudeBot, GPTBot, OAI-SearchBot, and 27 Other Bots Want From Your Website

Sooner or later every site owner finds one in the access logs: a user agent called ClaudeBot or GPTBot or OAI-SearchBot, hitting pages at 3 a.m. from an IP you've never seen. Three questions follow immediately. What is this thing? What is it doing with my content? And should I block it?

The answers matter more than most robots.txt decisions ever have, because these bots are the supply chain for AI answers. Whether ChatGPT, Claude, Perplexity, or Google's AI Overviews can cite your brand depends on which of these crawlers you let in — and one stale Disallow line can quietly remove you from a channel your competitors are growing in. This is the reference we wished existed: every AI crawler worth knowing in 2026, what it does, whether it behaves, and a sane framework for the block-or-allow call.

The three jobs an AI crawler can have

Vendor names make the bot landscape look complicated. Jobs make it simple. Every AI crawler on this page does one of three things:

1. Training collection. The bot gathers pages that may be used to train future models. Blocking it is a statement about training consent. It has little effect on whether AI products can cite you today. GPTBot, ClaudeBot, Meta-ExternalAgent, and CCBot live here.
2. Search indexing (retrieval). The bot builds the index an AI assistant searches at answer time. This is your citation pipeline. Block it and you're not in the candidate set when someone asks a question you should win. OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Bingbot live here.
3. User-triggered fetching. The bot fetches a specific URL because a live human asked the assistant to read it — closer to a browser visit than a crawl. ChatGPT-User, Claude-User, Perplexity-User, and Meta-ExternalFetcher live here.

Keep the three jobs in mind and every entry below becomes a one-line decision instead of a research project. The short version of the framework: retrieval and user-triggered bots are your distribution; training bots are your policy call.

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

OpenAI runs the cleanest three-way split, one bot per job, all documented on its crawler overview page.

GPTBot is the training crawler. Disallowing it signals your content shouldn't train OpenAI's foundation models. It respects robots.txt, and OpenAI publishes the exact IP ranges it crawls from at openai.com/gptbot.json.

OAI-SearchBot is the search-index crawler behind ChatGPT's web answers — the bot that decides whether ChatGPT can link to and cite you. It is explicitly not used for training. If AI citations are a channel you want, this is among the last user agents on earth you should block. IP ranges: openai.com/searchbot.json.

ChatGPT-User fires when a user pastes your URL into ChatGPT or asks it to read a page. Blocking it doesn't remove you from anything — it just makes your site fail when a real prospect actively tries to bring your content into their conversation, which is arguably the highest-intent visit an AI product can send you.

Anthropic: ClaudeBot, Claude-SearchBot, Claude-User

Anthropic mirrors the same three-job structure, and it's the source of the single most-searched bot name in this space. If ClaudeBot is what brought you here: it's Anthropic's training-collection crawler, it identifies itself with the user agent substring ClaudeBot and a contact address (+claudebot@anthropic.com), and per Anthropic's crawler documentation it honors robots.txt and won't attempt to bypass CAPTCHAs or other anti-circumvention measures.

Claude-SearchBot indexes content so Claude can return better search-grounded answers — Anthropic's citation pipeline, the analogue of OAI-SearchBot. Claude-User fetches pages when a Claude user asks about a specific site. Each of the three can be targeted independently in robots.txt, which means the same clean split is available: opt out of training with a ClaudeBot rule while leaving Claude-SearchBot and Claude-User free to keep you visible in Claude's answers.

Two housekeeping notes. You may still see Claude-Web or anthropic-ai in older block lists — legacy tokens from before Anthropic consolidated its fleet; harmless to keep, pointless to add today. And unlike OpenAI, Anthropic doesn't publish a verification IP list — its documented position is that robots.txt, not IP blocking, is the reliable control surface.

Perplexity: PerplexityBot and Perplexity-User

PerplexityBot builds the index behind Perplexity's answer engine. Perplexity states it isn't used for model training — indexing for citation is the whole job, which puts it squarely in the never-block column for anyone who wants AI visibility. Perplexity-User handles live user-requested fetches. Both publish IP ranges (perplexitybot.json, perplexity-user.json) and are documented on Perplexity's crawler page.

One asterisk that belongs in any honest reference: in August 2025 Cloudflare published evidence of undeclared crawling from Perplexity that circumvented robots.txt beyond these listed bots. Perplexity disputed the characterization. The practical takeaway isn't "panic-block Perplexity" — it's that robots.txt is a preference you publish, not a lock, and log-level observability (more on that below) is the only way to know what's actually reading you.

The impostors that aren't crawlers: Google-Extended and Applebot-Extended

These two cause more confusion than any real bot. Google-Extended never visits your site. It's a robots.txt control token: regular Googlebot does the crawling it always did, and a User-agent: Google-Extended rule tells Google whether that already-fetched content may be used for Gemini training and grounding. Blocking it does not remove you from Google Search — and, importantly, it doesn't control AI Overviews either, which ride on the normal search index. Applebot-Extended works the same way for Apple: Applebot crawls (it always has, for Siri and Spotlight), and the -Extended token only governs use of that content for Apple's foundation models.

So the scary-sounding advice "block Google-Extended and Applebot-Extended" costs you nothing in crawl traffic and changes nothing about your search presence. It's purely a training-consent lever — set it by policy, not by fear.

Bingbot: the AI crawler nobody thinks of as one

Bingbot predates the AI era, which is exactly why it's dangerous to mishandle. Bing's index feeds Microsoft Copilot end to end and remains a retrieval source in the ChatGPT ecosystem. A robots.txt that blocks Bingbot — often a leftover from a decade-old "we only care about Google" decision — now silently removes you from multiple AI surfaces at once. If you audit nothing else after reading this, audit your robots.txt for a legacy Bingbot rule.

This is also why IndexNow punches above its weight for AI visibility: pinging Bing's index on publish is the fastest route into the retrieval layer that Copilot and ChatGPT lean on.

The rest of the fleet

The long tail, by job:

Training collectors. CCBot is Common Crawl — a nonprofit whose corpus feeds many labs' training sets, so one rule covers many downstream models. Meta-ExternalAgent is Meta's AI training crawler (FacebookBot is its older sibling; Meta-ExternalFetcher is the user-triggered one). cohere-ai collects for Cohere. AI2Bot serves the Allen Institute's open models. Bytespider is ByteDance's — the fleet's problem child, widely documented ignoring robots.txt entirely; if you're serious about excluding it, you'll need server-level rules, not polite requests.

Answer engines and assistants. Amazonbot feeds Alexa and Amazon's shopping AI. DuckAssistBot powers DuckDuckGo's AI answers. YouBot is You.com. PetalBot is Huawei's Petal Search. iaskspider is iAsk.ai. MistralAI-User is Mistral's user-triggered fetcher for Le Chat. TimpiBot crawls for the Timpi decentralized index.

Services. Diffbot extracts structured data for enterprise customers; AwarioRssBot and AwarioSmartBot power brand-monitoring feeds. Not citation pipelines, rarely worth a rule either way.

Block or allow: the actual decision

Run each bot through its job, and the policy mostly writes itself:

• Allow the retrieval layer (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, DuckAssistBot, Amazonbot) unless you have a specific business reason to be invisible in that engine's answers. These bots are how AI citations happen.
• Allow the user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, Meta-ExternalFetcher). Blocking them only breaks the moment a real person tries to hand your content to their assistant.
• Decide training collectors on policy (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, cohere-ai, AI2Bot). Publishers with licensing leverage increasingly block these while keeping retrieval open; most small-business sites conclude the marginal training contribution costs them nothing and leave everything open. Both are defensible. What's not defensible is blocking them by accident with a User-agent: * rule that takes the retrieval bots down too.
• Treat Bytespider separately, since robots.txt won't stop it anyway.

The pattern to avoid — and the one we see constantly in robots.txt audits — is the blanket AI block someone added in 2023 from a copy-paste list, which is still silently vetoing every AI channel the business now wants. The mirror image works: a deliberate robots.txt that names each bot is a competitive asset precisely because most sites never make the distinction.

Trust, but verify the IP

One more thing the copy-paste lists never mention: user-agent strings are free to fake. Aggressive scrapers routinely borrow GPTBot's name because site owners allow it. When a bot's behavior looks wrong — crawl volume way above what your site's size justifies, requests hammering paths no legitimate indexer wants — check the source IP before drawing conclusions. OpenAI and Perplexity publish per-bot CIDR files (linked above) exactly for this; a "GPTBot" request from an IP outside gptbot.json is not GPTBot, and blocking that costs you nothing.

First, find out who's actually visiting you

Everything above is the map. Your server logs are the territory, and they diverge more often than you'd think: sites convinced they're being ignored by AI turn out to have blocked the one bot that mattered; sites worried about scraping discover the "AI bot problem" is one impostor UA from a hosting subnet.

Google Search Console can't help here — it doesn't log a single one of these bots. The free AuditAE WordPress plugin closes the gap in one activation: it recognizes every user agent in this post, logs each hit locally in a lightweight table (nothing leaves your site), and shows per-bot counts, top-crawled paths, and a 🎉 first-visit alert when a new engine discovers you. Its robots.txt audit cross-checks your live file against the citation-critical list — using the same group-matching logic real crawlers use — and a one-click fix appends an explicit Allow group if something important is walled out. The verdict feeds your site's AI Readiness Score, so a robots.txt regression shows up as a scored, explained point drop instead of a silent traffic mystery.

Track the bots first, then set policy from data. The block-or-allow call gets easy once you can see who's actually knocking.