AuditAE-Bot fetches a site
only when someone asks.
If you found this URL in your server logs, a person or their AI assistant asked AuditAE to check your site: whether AI crawlers can reach it, whether its llms.txt is valid, or whether AI engines can read and cite it. Here’s what we request, how often, and how to reach us.
Questions or a problem with our traffic: support@auditae.app
How to recognise us.
AuditAE-Bot
Every ordinary request: pages, robots.txt, sitemaps and llms.txt.
Mozilla/5.0 (compatible; AuditAE-Bot/1.0; +https://auditae.app/about-bot)AuditAE-Probe
The AI crawler checker loads the homepage once with each of these, to see whether a firewall or CDN answers AI bots differently from a normal visitor, and the agent readiness check loads it once as ChatGPT-User. Each carries the vendor’s bot token plus our suffix, so it can’t be mistaken for the real crawler.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot AuditAE-Probe (+https://auditae.app/about-bot)Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) AuditAE-Probe (+https://auditae.app/about-bot)Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) AuditAE-Probe (+https://auditae.app/about-bot)Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot AuditAE-Probe (+https://auditae.app/about-bot)
One check, a handful of requests.
- Free AI search audit
- robots.txt, /llms.txt, the sitemap, the page entered and up to four more pages from the sitemap or navigation.
- AI crawler checker
- robots.txt and the homepage, plus the four-request firewall probe described below.
- llms.txt generator and validator
- The homepage, robots.txt and /llms.txt, then either the sitemap and up to 20 pages for their titles and descriptions, or /llms-full.txt and up to 10 links from the file to check they resolve.
- Citation audits and fix plans
- The homepage and a few key pages, robots.txt, /llms.txt and the sitemap, to read what the site already says and serves before recommending changes.
- Agent readiness check
- For AI assistants connected to AuditAE's MCP server: the homepage twice (once as AuditAE-Bot, once as ChatGPT-User with the AuditAE-Probe suffix, to spot a bot wall), robots.txt, five /.well-known/ files (MCP server cards, agent card, UCP and ACP), and a pricing and a product page when the homepage links to them.
- Technical site audits
- A crawl of the site's own pages, requested by a signed-in user. It follows the User-agent: * rules in your robots.txt and stays on one host.
- Plain HTTP requests, no JavaScript, no cookies, no logins and no form submissions.
- Public web addresses on ports 80 and 443 only, with a short timeout and a cap on how much of each response we read.
- The AI crawler checker and the llms.txt tool share a limit of 30 runs per checked site per hour. A repeat run is usually answered from a short-lived cache; a manual re-run fetches again.
Questions
Someone asked AuditAE to check your site: a visitor using one of the free tools, a customer auditing their own or a client's site, or an AI assistant connected to AuditAE's MCP server. It doesn't crawl the web on its own schedule.
No. It fetches pages to answer one check and doesn't build a search index.
That's the AI crawler checker's firewall probe. It loads your homepage with each vendor's user-agent to see whether your CDN or firewall answers those bots differently from a normal visitor. The AuditAE-Probe suffix is there so you can tell it apart from the real bot: the real GPTBot never carries it.
Only as often as people run checks on it. The AI crawler checker and the llms.txt tool share a limit of 30 runs per checked site per hour, and a repeat run within the cache window (10 minutes for the crawler checker, 30 for the llms.txt tool) is usually answered from cache without contacting your site. A manual re-run fetches again, still within the 30-per-hour limit. The firewall probe only runs when that per-site limit can be enforced; if it can't, the probe is skipped.
Match "AuditAE-Bot" or "AuditAE-Probe" in the User-Agent header in your firewall or CDN rules. If something about our traffic looks wrong, email support@auditae.app with your domain and what you saw in your logs.