AI Crawlers Explained: What ClaudeBot, GPTBot, OAI-SearchBot, and 27 Other Bots Want From Your Website

Sooner or later every site owner finds one in the access logs: a user agent called ClaudeBot or GPTBot or OAI-SearchBot, hitting pages at 3 a.m. from an IP you've never seen. Three questions follow immediately. What is this thing? What is it doing with my content? And should I block it?

The answers matter more than most robots.txt decisions ever have, because these bots are the supply chain for AI answers. Whether ChatGPT, Claude, Perplexity, or Google's AI Overviews can cite your brand depends on which of these crawlers you let in, and one stale Disallow line can quietly remove you from a channel your competitors are growing in. This post covers every AI crawler worth knowing in 2026: what it does, whether it behaves, and a sane framework for the block-or-allow call.

The three jobs an AI crawler can have

Vendor names make the bot list look complicated. Jobs make it simple. Every AI crawler on this page does one of three things:

1. Training collection. The bot gathers pages that may be used to train future models. Blocking it is a statement about training consent. It has little effect on whether AI products can cite you today. GPTBot, ClaudeBot, Meta-ExternalAgent, and CCBot live here.
2. Search indexing (retrieval). The bot builds the index an AI assistant searches at answer time. This is your citation pipeline. Block it and you're not in the candidate set when someone asks a question you should win. OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Bingbot live here.
3. User-triggered fetching. The bot fetches a specific URL because a live human asked the assistant to read it. It's closer to a browser visit than a crawl. ChatGPT-User, Claude-User, Perplexity-User, and Meta-ExternalFetcher live here.

Keep the three jobs in mind and every entry below becomes a one-line decision instead of a research project. The short version of the framework: retrieval and user-triggered bots are your distribution; training bots are your policy call.

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

OpenAI runs the cleanest three-way split, one bot per job, all documented on its crawler overview page.

GPTBot is the training crawler. Disallowing it signals your content shouldn't train OpenAI's foundation models. It respects robots.txt, and OpenAI publishes the exact IP ranges it crawls from at openai.com/gptbot.json.

OAI-SearchBot is the search-index crawler behind ChatGPT's web answers: the bot that decides whether ChatGPT can link to and cite you. It is explicitly not used for training. If AI citations are a channel you want, don't block it. IP ranges: openai.com/searchbot.json.

ChatGPT-User fires when a user pastes your URL into ChatGPT or asks it to read a page. Blocking it doesn't remove you from anything. It makes your site fail when a real prospect actively tries to bring your content into their conversation, which is arguably the highest-intent visit an AI product can send you. Worth knowing before you write the rule: OpenAI documents that because these fetches are initiated by a person, "robots.txt rules may not apply." Treat a ChatGPT-User Disallow as a stated preference, not a guarantee. If you need to stop those fetches, do it at the server or WAF layer.

Anthropic: ClaudeBot, Claude-SearchBot, Claude-User

Anthropic mirrors the same three-job structure, and it's the source of the single most-searched bot name in this space. If ClaudeBot is what brought you here: it's Anthropic's training-collection crawler, it identifies itself with the user agent substring ClaudeBot and a contact address (+claudebot@anthropic.com), and per Anthropic's crawler documentation it honors robots.txt and won't attempt to bypass CAPTCHAs or other anti-circumvention measures.

Claude-SearchBot indexes content so Claude can return better search-grounded answers. It's Anthropic's citation pipeline, the analogue of OAI-SearchBot. Claude-User fetches pages when a Claude user asks about a specific site. Each of the three can be targeted independently in robots.txt, which means the same clean split is available: opt out of training with a ClaudeBot rule while leaving Claude-SearchBot and Claude-User free to keep you visible in Claude's answers.

Two housekeeping notes. You may still see Claude-Web or anthropic-ai in older block lists. They're legacy tokens that Anthropic's current documentation no longer lists: harmless to keep, pointless to add today. Anthropic now publishes its bots' IP ranges at claude.com/crawling/bots.json, but its documented position is still that robots.txt, not IP blocking, is the reliable control, because blocking IPs can stop its bots from reading your robots.txt at all.

Perplexity: PerplexityBot and Perplexity-User

PerplexityBot builds the index behind Perplexity's answer engine. Perplexity states it isn't used for model training. Indexing for citation is the whole job, which puts it squarely in the never-block column for anyone who wants AI visibility. Perplexity-User handles live user-requested fetches. Both publish IP ranges (perplexitybot.json, perplexity-user.json) and are documented on Perplexity's crawler page.

One asterisk that belongs in any honest reference: in August 2025 Cloudflare published evidence of undeclared crawling from Perplexity that circumvented robots.txt beyond these listed bots. Perplexity disputed the characterization. The practical takeaway isn't "panic-block Perplexity". It's that robots.txt is a preference you publish, not a lock, and log-level observability (more on that below) is the only way to know what's actually reading you.

The impostors that aren't crawlers: Google-Extended and Applebot-Extended

These two cause more confusion than any real bot. Google-Extended never visits your site. It's a robots.txt control token: regular Googlebot does the crawling it always did, and a User-agent: Google-Extended rule tells Google whether that already-fetched content may be used for Gemini training and grounding. Blocking it does not remove you from Google Search, and it doesn't control AI Overviews either, which ride on the normal search index. (For that, Search Console has offered a site-level opt-out from AI Overviews and AI Mode since August 2026; Google says it isn't a ranking signal for regular results.) Applebot-Extended works the same way for Apple: Applebot crawls (it always has, for Siri, Spotlight and Safari), and the -Extended token only governs use of that content for Apple's foundation models.

So the scary-sounding advice "block Google-Extended and Applebot-Extended" costs you nothing in crawl traffic and changes nothing about your search presence. It's purely a consent lever for training (and, for Google, Gemini grounding), so set it by policy.

Bingbot: the AI crawler nobody thinks of as one

Bingbot predates the AI era, which is exactly why it's dangerous to mishandle. Bing's index feeds Microsoft Copilot end to end and remains a retrieval source in the ChatGPT ecosystem. A robots.txt that blocks Bingbot (often a leftover from a decade-old "we only care about Google" decision) now silently removes you from multiple AI surfaces at once. If you audit nothing else after reading this, audit your robots.txt for a legacy Bingbot rule.

This is also why IndexNow punches above its weight for AI visibility: pinging Bing's index on publish is a quick way into the retrieval layer that Copilot uses and that ChatGPT draws on in part.

The rest of the fleet

The long tail, by job:

Training collectors. CCBot is Common Crawl, a nonprofit whose corpus feeds many labs' training sets, so one rule covers many downstream models. Meta-ExternalAgent is Meta's AI training crawler (FacebookBot is its older sibling; Meta-ExternalFetcher is the user-triggered one). cohere-ai still shows up in block lists, though Cohere currently lists no active crawlers. AI2Bot serves the Allen Institute's open models. Amazonbot is Amazon's general crawler, and Amazon says what it collects may be used to train Amazon AI models. Bytespider is ByteDance's, widely reported to ignore robots.txt entirely; if you're serious about excluding it, you'll need server-level rules, not polite requests.

Answer engines and assistants. DuckAssistBot powers DuckDuckGo's AI answers. YouBot is You.com. PetalBot is Huawei's Petal Search. iaskspider is iAsk.ai. MistralAI-User is Mistral's user-triggered fetcher for Le Chat. TimpiBot crawls for the Timpi decentralized index.

Services. Diffbot extracts structured data for enterprise customers; AwarioRssBot and AwarioSmartBot power brand-monitoring feeds. Not citation pipelines, rarely worth a rule either way.

Block or allow: the actual decision

Run each bot through its job, and the policy mostly writes itself:

• Allow the retrieval layer (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, DuckAssistBot) unless you have a specific business reason to be invisible in that engine's answers. These bots are how AI citations happen.
• Allow the user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, Meta-ExternalFetcher). Blocking them only breaks the moment a real person tries to hand your content to their assistant, and it may not even work, since OpenAI and Perplexity both document that robots.txt may not apply to fetches a human explicitly requested.
• Decide training collectors on policy (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, cohere-ai, AI2Bot, Amazonbot). Many large news publishers block these, some while keeping the retrieval bots open; plenty of smaller sites decide the marginal training contribution costs them nothing and leave everything open. Both are defensible. What's not defensible is blocking them by accident with a User-agent: * rule that takes the retrieval bots down too.
• Treat Bytespider separately, since robots.txt won't stop it anyway.

The pattern to look for in a robots.txt audit is a blanket AI block someone added in 2023 from a copy-paste list, which is still silently vetoing every AI channel the business now wants. The mirror image works: a deliberate robots.txt that names each bot is a competitive asset precisely because many sites never make the distinction. To see where yours stands, the free AI crawler checker gives every bot in this post a verdict from your live robots.txt, with the line that decided it.

Trust, but verify the IP

One more thing the copy-paste lists never mention: user-agent strings are free to fake. Aggressive scrapers routinely borrow GPTBot's name because site owners allow it. When a bot's behavior looks wrong (crawl volume way above what your site's size justifies, requests hammering paths no legitimate indexer wants), check the source IP before drawing conclusions. OpenAI, Perplexity and Anthropic publish CIDR files (linked above) for this; a "GPTBot" request from an IP outside gptbot.json is not GPTBot, and blocking that costs you nothing.

First, find out who's actually visiting you

Everything above is the map. Your server logs are the territory, and the two can diverge: a site convinced it's being ignored by AI may turn out to have blocked the one bot that mattered; a site worried about scraping may find the "AI bot problem" is one impostor UA from a hosting subnet.

Google Search Console can't help here: it doesn't log any of these bots. The free AuditAE WordPress plugin closes the gap in one activation: it recognizes every crawler profiled in this post that actually visits, logs each hit locally in a lightweight table (nothing leaves your site), and shows per-bot counts, top-crawled paths, and a 🎉 first-visit alert when a new engine discovers you. Its robots.txt audit cross-checks your live file against the citation-critical list, using the same group-matching logic real crawlers use. If something important is walled out, a one-click fix appends an explicit Allow group to WordPress's virtual robots.txt (sites with a physical robots.txt file get the lines to paste instead). The verdict feeds your site's AI Readiness Score, so a robots.txt regression shows up as a scored, explained point drop instead of a silent traffic mystery.

Track the bots first, then set policy from data. The block-or-allow call gets easy once you can see who's actually knocking.

Not sure where your own site stands on these checks? The free AI search audit scores any domain 0–100 on robots.txt bot rules, llms.txt, Organization and Article schema, snippet directives, server-rendered text and freshness, and emails the fix for each one. No login, ten seconds.

Want the verdict for your own site? The free AI crawler checker checks your robots.txt against every major AI crawler, quotes the line that decides each one, and tests whether your firewall turns GPTBot, ClaudeBot, PerplexityBot or ChatGPT-User away. Your first check needs no login.