Free AI crawler checker: is your site
blocking AI bots?

Your robots.txt names the bot by part of its name, for example “User-agent: claude” rather than “User-agent: ClaudeBot”. Crawlers that match partial names would apply that group; crawlers that follow the robots.txt standard (RFC 9309) match the full name and fall back to the “User-agent: *” group. When those two readings give different answers, we show Unclear with both lines instead of guessing. Name the bot in full to remove the doubt. For the bots it scores, our AI readiness audit counts a partial-name block as blocked, to be safe.

The bot can reach your site but is disallowed from some paths, such as /wp-admin/ or /private/. That's usually deliberate. Use the path test under the form to check one page for each AI and search bot.

Robots.txt is a request; a CDN or firewall rule is enforcement. We load your homepage once as a normal request and four more times with the user-agents of GPTBot, ClaudeBot, PerplexityBot and ChatGPT-User. If those get a 401, 403, 429, 503 or a challenge page while the normal request didn't, something between the bots and your site is stopping them. We send the user-agent but not the vendor's IP addresses, and some CDNs let the real bots through while stopping look-alikes, so the result is a strong hint, not proof. Check your CDN's bot settings to confirm.

We follow RFC 9309, the robots.txt standard. A 404, 410 or other 4xx means there are no rules, so every bot is allowed. A 5xx server error means crawlers treat the whole site as blocked until it recovers, so we show that in red. A 401, 403 or 429 is an answer to our request (HTTP auth, a firewall or a bot rule), so we can't tell what AI crawlers see and say so. A web page served at /robots.txt counts as no rules.

Blocking a training crawler such as GPTBot or ClaudeBot, or the Google-Extended token, is a policy choice about model training, not a visibility problem, so it never turns the result red. The headline counts only the 14 AI and search bots that fetch pages for AI search and for people's questions. Training blocks are shown as a neutral count.

Use the “Block training, keep AI search access” fix in the result: it disallows the training bots and allows the search and user-triggered ones, keeping your existing path rules. The cost: training data is also how AI learns your brand, so blocking it can reduce mentions of you that appear without a link. Run the free citation check to see whether AI mentions you before you decide.

Check whether AI mentions you →

Not necessarily. Access is the precondition, not the result. Whether ChatGPT, Perplexity, Gemini or Google AI Overviews cite you depends on your content, what else answers the question and which sources each engine trusts. The free citation check runs real prompts through the engines and shows who they cite.

Run the free citation check →

We cache the result for up to 10 minutes (1 minute if the firewall check was skipped) so a re-check is instant, and log the domain checked and summary counts. The domain is also kept for an hour as a rate-limit key, so no single site gets checked too often, and your IP address is kept for 24 hours for the per-visitor limit. We don't store the page content.

Three OpenAI user-agents with three jobs: training, search, and fetching a page a person asked about. The crawler guide explains each one.

AI crawlers explained →

OpenAI documents GPTBot as its training crawler and OAI-SearchBot as the one that surfaces sites in ChatGPT search, so they're separate decisions. The crawler guide covers the details.

AI crawlers explained →