Using AuditAE

Site Audit: crawl a site and read Site Health

Crawl up to 10,000 pages for 1¢ per 10 pages and get Site Health, 9 theme scores and 173 checks, each with why it matters and how to fix it.

Updated Oct 13, 2026View as Markdown

On this page· 15 sections

Steps

1. Start a crawl

Open a site's Technical tab from your sites and click Crawl site (or Re-crawl site). Pick how many pages to crawl, up to 10,000; the most it can cost is shown before you start. An agent can start the same crawl with crawl_site over MCP, and a script with POST /api/v1/crawl over the REST API.

2. Read Site Health and the themes

When the crawl finishes, the tab shows Site Health (0–100), the number of errors, warnings and notices, and a score for each of the 9 themes. Checks are listed worst first: each one says how many items failed out of how many were checked, why it matters and how to fix it. Core Web Vitals show as Pending for a few minutes after the crawl, while they're measured.

3. Open a check and fix it

Click View pages on a check to see every affected URL with its evidence (the status code, the duplicate title, the broken link and the pages that link to it). Checks AEBOT can help with have a Fix with AEBOT button. Agents read the same list with get_crawl_issue, free.

4. Schedule it

To keep watching the site, turn on Scheduled crawls on the same tab: weekly or monthly, with a summary email after each run. Setting it up is free; each run bills 1¢ per 10 pages when it runs.

5. Crawl again and compare

Re-crawl once your fixes are live. Each new crawl is compared with the one before it: the Site Health card shows the change ("+4 since Oct 5 · 12 new · 30 fixed"), each check shows how many issues are new and how many were fixed, and Compare lists every change. Reading and comparing past crawls is free.

A Site Audit crawls a whole site and grades it the way Semrush's Site Audit does: 173 checks (47 errors, 62 warnings and 64 notices) across 9 themes, a Site Health score and a score per theme. The checks cover 92 of Semrush's Site Audit issues, plus on-page, content and AI search checks Semrush doesn't have. For a single page with Core Web Vitals instead, use the free homepage technical audit on the same tab (or run_technical_audit).

How a crawl runs

Seed and site checks. The crawl starts at the homepage, reads robots.txt and the sitemaps (up to 10,000 sitemap URLs), and queues them. At the same time it runs the site checks: whether HTTP redirects to HTTPS, whether www and non-www end up on one host, the TLS certificate (expiry, host name, trust) and HTTP/2, whether an unknown URL answers 404, llms.txt, and whether a firewall blocks AI fetchers.
Pages. It fetches pages about 3–6 a second, follows every internal link (www and non-www count as one site) and obeys robots.txt's User-agent: * rules. Each page is checked as it's fetched: status, redirects, title, description, headings, canonical, hreflang, schema, images, links, speed and more.
Links, images and scripts. It then checks up to 5,000 of the files and external links the pages point to (5 per page crawled, at least 500, at most 20 per external site): broken images, scripts and stylesheets, broken external links, images over 200 KB, and whether JS and CSS files are compressed, cached and minified.
Scoring. Once that's done, the checks that compare pages run: duplicate titles, descriptions and content, broken internal links with the pages that link to them, orphan and single-link pages, canonical and hreflang targets, sitemap coverage, robots.txt and sitemap health. Then Site Health, the theme scores and each page's Internal LinkRank are worked out.
Core Web Vitals. A few minutes after the crawl completes, Google PageSpeed Insights measures up to 10 pages on mobile: the homepage, then the page with the highest Internal LinkRank in each section of the site (one page per 10 crawled). Until then the Core Web Vitals theme shows Pending.

A small site finishes while you wait. A bigger one keeps crawling in the background and the tab shows its progress; over MCP or REST a big crawl answers status: "running" and you read it later. While a crawl is still running, its counts are provisional.

What it costs

1¢ per 10 pages fetched: $0.10 for 100 pages, $1.00 for 1,000 and $10.00 for 10,000. It's billed as the crawl runs, so a crawl that runs out of credit stops as partial and is still scored on the pages it reached. The checks, Site Health, the theme scores and the reports are included, and so are the link, image and script checks (up to 5,000 per crawl), the site checks and the Core Web Vitals measurements (up to 10 pages): only pages fetched are billed. Reading and comparing crawls is always free: the dashboard, get_latest_crawl, get_crawl_issue, compare_crawls and the REST endpoints. Over REST a crawl covers 25 pages unless you set max_pages (up to 10,000). See Credits, prices and top-ups.

Comparing crawls

When a crawl finishes it's compared with the site's previous scored crawl, and the comparison is saved with it. It shows:

Site Health and the theme scores, before and now.
New and fixed issues per check. For a page check, new counts the pages failing it now that didn't before, and fixed the pages that failed before and don't now (a page that now errors or redirects isn't checked, so its old issues count here; see became broken below). A page the new crawl didn't reach counts as removed, not fixed. A site-wide check counts 1 when it starts or stops failing. For links, images and scripts it's the change in the count, and only new ones are listed, because a file that passes isn't stored.
Pages added, removed, whose status code changed, that became broken, noindex or indexable, and whose title changed. When either crawl stopped at its page limit, pages added and removed aren't counted (the crawl didn't see the whole site), and new issues can include pages the other crawl didn't reach: the Technical tab leaves the new counts out then, and vs_previous and get_changes add incomplete_baseline.
robots.txt: whether it changed and which User-agent: * rules were added or removed; and the sitemap URL count.

Only crawls graded with the same version of the checks are diffed page by page. When the checks change between two crawls, the comparison shows the totals only, without new and fixed counts or a Site Health change, because the scores were worked out with different rules. Crawls made before Site Audit checks existed are never compared, and a site's first scored crawl is the baseline. Core Web Vitals are measured a few minutes after a crawl, so their theme change appears then.

On the Technical tab, Compare shows the comparison, and Crawl history lists the last 12 crawls with their Site Health and change; pick any earlier crawl there to compare with it. Comparing is free and fetches nothing.

Scheduled crawls

A crawl can repeat on its own, weekly on a day you pick or monthly on a day from 1 to 28. Turn it on in the Scheduled crawls card on the Technical tab (pick Repeat, the day, the most pages per run and whether to email you a summary), or ask an agent with schedule_crawl. Daily isn't offered: a 10,000-page site crawled every day would cost about $300 a month.

When it runs. Each site gets a fixed hour between 05:00 and 10:00 UTC (spread by site, so they don't all start at once). If a crawl of the site is already running then, the scheduled one waits an hour.
What each run does. It crawls up to its page limit like any crawl, is scored, and is compared with the crawl before it (see Comparing crawls above). With the email on, you get a summary afterwards: Site Health and its change, errors, warnings and notices with their change, up to 5 checks with new problems and 5 with fixes, the pages crawled and what the run cost, with a link straight to the comparison. A crawl you start from the dashboard that keeps running in the background after you leave sends the same summary.
What it costs. Setting up, changing and stopping a schedule is free. Each run bills 1¢ per 10 pages as it crawls, like any crawl. The card (and schedule_crawl) shows the most a run can cost and about what it comes to a month at your last whole-site crawl's size (a URL list's size for a list schedule). The default page limit is your last whole-site crawl's size plus 20%, rounded up to the hundred, so a growing site isn't cut short; a URL-list schedule defaults to the list's length plus the homepage. Changing one setting over the API, MCP or AEBOT keeps the others, and keeps the schedule's URL list unless you pass a new list or another source.
When it pauses. A run that finds your balance at zero pauses the schedule and you get a low-balance email; top up and press Resume. A schedule also pauses when its site is archived or the workspace is closed. Choose Off to stop it; past crawls stay.

Crawl settings

Open Advanced settings under Crawl site to change what a crawl covers and how it fetches. Tick Save as this site's default to keep them for every later crawl and scheduled run (a call's own settings always win). Agents pass the same settings to crawl_site and schedule_crawl, and scripts to POST /api/v1/crawl; GET and PUT /api/v1/crawl/settings read and set the site's defaults. Left alone, a crawl covers the whole site exactly as described above.

What to crawl. The whole site (links and sitemap, the default); sitemaps only (the robots.txt Sitemap: files, else /sitemap.xml); one sitemap (a file or index on the site); or a list of URLs (up to 10,000 over the API, 2,000 per MCP call, 5,000 from the dashboard's box or a .txt / .csv file, first column). The homepage is always fetched too, because the site-wide checks read it, and a list crawl's page limit is the list's length plus the homepage. URLs off the site, files and repeats are rejected and listed back (the dashboard shows how many, with a few examples). With any source but the whole site, links on the crawled pages count as inlinks but aren't followed.
Only crawl / skip paths. Up to 20 include and 50 exclude masks, in robots.txt syntax matched against the path and query: * matches anything, $ ends the path, anything else is a prefix. /shoes also matches /shoes-men; use /shoes/ for the folder only. Skip beats only-crawl. A skipped URL is never fetched or billed and counts in skipped by rules. The start page ignores include rules, but a crawl whose start page matches a skip rule is refused.
Ignore query parameters. Names to drop from URLs (sort, page_*), so /shop?sort=price and /shop are one page; * drops every query string. Tracking parameters (utm_*, gclid, fbclid …) are always dropped.
Subdomains. Off by default (sites are added by their host). On, links to any subdomain of the site's domain are crawled too, each obeying its own robots.txt, up to 25 subdomains; links to more count as skipped by rules.
robots.txt rules to obey. By default the crawl obeys User-agent: *. Pick Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot or PerplexityBot to crawl the pages that bot's group allows (its own group, else *), to see the site as that bot may. The crawler still identifies as AuditAE-Bot; it never pretends to be another bot.
Device. Desktop (the default) or mobile: both are the AuditAE-Bot browser string.
Speed. Normal (3 pages at a time), polite (one page a second; a 10,000-page crawl then takes about 3 hours), or the robots.txt Crawl-delay (up to 10 seconds, else polite). The link and image checks also go one at a time to the site at the polite speeds.
Core Web Vitals pages. Up to 10 pages on the site to measure instead of the automatic sample.
Hide check. Each check card has Hide check: the check leaves Site Health and the theme scores of every crawl of the site at once (shown under Not checked as excluded, with Show again; a crawl that ran while the check was hidden, or with exclude_checks, keeps it excluded, and it is counted again from the next crawl). Agents and scripts pass exclude_checks.

What a narrowed crawl can't judge: with include or exclude rules, or any source but the whole site, orphan_sitemap_page, single_inlink, nofollow_only_inlinks, indexable_not_in_sitemap, deep_page need every link on the site, so they show as Not checked. A URL list also skips the sitemap joins (sitemap_bad_urls, sitemap_noindex, sitemap_non_canonical). The site checks, the link and image checks and Core Web Vitals run for every source.

Comparing: a crawl is compared with the previous crawl made with the same scope (source, URL list, rules, ignored parameters, subdomains and robots.txt group) when there is one. Two crawls of different scopes are still compared, but say the scope changed and leave out pages added and removed, because different settings reach different pages.

Site Health

Site Health starts at 100. Each error check that fails costs 2–4 points and each warning 0.5–1 point, more the larger the share of checked items failing it; notices and Core Web Vitals don't count. The score is rounded and never goes below 0.

So one broken page out of 800 costs an error a little over 2 points, while a DNS failure (a site-wide error) costs the full 4. Because AuditAE runs more checks than Semrush, with more site-level errors, the same site usually scores a few points lower here.

Theme scores

The 9 themes are Semrush's eight (Crawlability, HTTPS, International SEO, Core Web Vitals, Site Performance, Internal Linking, Markup and AI Search) plus On-page and content, an AuditAE extra for title, description, heading and content checks.

A theme's score is the weighted share of its checks that pass: errors weigh 4, warnings 1, notices ¼, and a failing check counts as half failed plus half the share of its items that fail. Checks that didn't run are left out, and a theme where nothing ran has no score. Two themes use Semrush's own rules: Markup is valid structured-data items ÷ all items, and Core Web Vitals is (good pages + ½ needs-improvement pages) ÷ pages measured.

Errors, warnings, notices and "not checked"

Errors stop pages being crawled, indexed or trusted (broken pages, DNS failures, redirect loops, pages blocked from search).
Warnings hurt rankings or AI citations but don't block them (missing meta descriptions, nofollow internal links, content that needs JavaScript).
Notices are worth knowing but often deliberate (deep pages, nofollow external links, no llms.txt). They move theme scores, never Site Health.
Not checked means the crawl didn't collect what a check needs, with the reason: for example, the crawl stopped at its page limit, so inlink counts are incomplete. Not-checked checks never count against you.

Links, site checks and Core Web Vitals

Links, images and scripts (11 checks). Each file or external link gets one light request (a HEAD, or a GET when the server refuses HEAD), following up to 5 redirects. It counts as broken only on a 404 or 410, a server error (except 503, 508 and 509, which mean an overloaded or rate-limiting server, and Cloudflare's 520–527), or a host that doesn't exist or refuses the connection. Other answers (401, 403, 429, other 4XX codes, a bot challenge page), timeouts, dropped connections, TLS errors and failed DNS lookups are left unverified, never broken; external links answering 403 get their own notice. Up to 200 of the site's own JS and CSS files are read in part to judge minification. Links past the per-crawl limit aren't checked.
Site checks. A few requests, made once per crawl: the HTTP homepage, the www or non-www twin, a made-up URL that should return 404, /llms.txt, a TLS handshake that reads the certificate and whether HTTP/2 is offered, and the homepage requested as ChatGPT-User (with an AuditAE-Probe suffix so site owners can tell it apart) to see whether a firewall blocks AI fetchers. check_llms_txt and check_ai_crawlers give more detail on the last two, free.
Core Web Vitals (5 checks). Lab LCP, CLS and Total Blocking Time from PageSpeed Insights on mobile; INP only when Google has field data for the page or the site. They move the Core Web Vitals theme only, never Site Health. A crawl with no page that answered (or a PageSpeed outage) shows them under Not checked. For one page on mobile and desktop, use the free homepage technical audit.

Crawls run before these checks existed list them under Not checked, which never affects a score.

How AuditAE differs from Semrush

User agent. The crawler identifies as AuditAE-Bot with a desktop browser string by default (Semrush defaults to mobile; pick mobile under Advanced settings), and page crawling obeys robots.txt's User-agent: * rules unless you pick another bot's group. The once-per-crawl site checks (the http:// and www/non-www homepage, a made-up 404 URL, /llms.txt and the TLS handshake) and the single light request to each file or outbound link don't consult robots.txt; an outbound link's own site's robots.txt is never read. The only request not sent as AuditAE-Bot is the AI-fetcher check: one homepage request as ChatGPT-User, skipped when robots.txt blocks ChatGPT-User or can't be read.
Subdomains aren't crawled by default (Semrush crawls them for a root domain), because a site is added by its host; turn them on under Advanced settings. www and non-www are treated as the same site.
Stricter severities. Blocking AI search bots site-wide (ai_search_bots_blocked) and blocking Googlebot or Bingbot site-wide (search_bots_blocked) are errors.
An extra theme, On-page and content, and AI search checks for llms.txt, AI bot access in robots.txt and Content-Signal.
No JavaScript rendering. Pages are read as the server sends them, as most AI crawlers read them; a page that only renders in the browser fails js_dependent_content.

How AuditAE differs from Screaming Frog

Screaming Frog's SEO Spider is a desktop app licensed per user per year, free up to 500 URLs. The Site Audit runs in the cloud and bills 1¢ per 10 pages crawled, with nothing to install.

Screaming Frog only: JavaScript rendering, crawls over 10,000 URLs, custom extraction and custom JavaScript, crawling behind a login or basic auth, private staging or localhost, ignoring robots.txt, log files, and Google Analytics and Search Console data joined to each URL.
Not on Screaming Frog's feature list: scheduled crawls that run with no computer left on, a hosted MCP server an agent uses to start, compare and schedule crawls (crawl_site, compare_crawls, schedule_crawl), Site Health with 9 theme scores, the AI search checks, and free team seats on one balance.

Prices, a cost table and when to keep Screaming Frog: Screaming Frog alternative.

Not built (yet)

Not offered: crawling pages robots.txt disallows, password-protected (basic-auth) sites and Web Bot Auth (sites aren't ownership-verified: anyone can add any domain), and JavaScript rendering. These Semrush checks aren't built yet:

tls_old_protocol (Semrush 28): Old TLS versions (1.0, 1.1). Needs a deliberately downgraded handshake our runtime can't make reliably yet.
tls_weak_cipher (Semrush 42): Weak TLS ciphers. Same reason: needs a deliberately weak handshake.
tls_no_sni (Semrush 127): TLS without SNI. A server that doesn't support SNI already shows up as a certificate name mismatch.
orphan_ga (Semrush 206): Orphan pages found in Google Analytics. Needs GA4 page paths joined per site; planned as a follow-up.
high_document_interactive (Semrush 208): Time to Interactive. Needs page rendering, and Lighthouse 10 dropped the metric.
external_resources_blocked (Semrush 210): Resources blocked by another host's robots.txt. Would mean fetching robots.txt from every external host a page loads from.
amp_invalid: AMP validation. Needs the AMP validator.
internal_link_rank: Internal LinkRank is shown as a page metric (0–100), not as a pass/fail check.

For agents and the API

crawl_site starts a crawl and returns Site Health, the theme scores and the failing checks (each with an id, severity, title and "n of m"). Its optional settings (source, urls, include, exclude, ignore_params, subdomains, robots_agent, device, speed, cwv_urls, exclude_checks, save_as_default) are described above; the answer adds settings, skipped_by_rules and, for a list, rejected. cwv_state: "pending" means Core Web Vitals are still being measured; read the crawl again a few minutes later.
get_latest_crawl reads the latest crawl, or an older one by run_id, free. Pass theme for that theme's report (load-time buckets, depth and inlink buckets, hreflang coverage, AI bot access and so on), severity to filter, or check to list the pages failing one check.
get_crawl_issue reads one check: its rule, why it matters, how to fix it, which tools fix it, and the affected URLs with evidence, 25 at a time. Free.
optimize_image fixes large_images: with site_id it takes the crawl's images over 200 KB, largest first, resizes them (1920 px wide by default, never enlarging), compresses them to WebP (keeping their format on WordPress, for /wp-content/uploads/ and next/image sources, or with format: "original") and steps them down toward 200 KB without visible loss. Each comes back with a private download link valid for 48 hours, the size before and after, and a paste-ready <img> tag. It also takes urls, for images on a CDN the crawl counts as external. AEBOT has it too (ask it to "shrink the large images"), and POST /api/v1/images/optimize returns the same answer over REST. 1¢ per 5 images optimized; an image that barely shrinks, a skipped or failed one and a repeat within 24 hours are free. It doesn't change the site: swap the files in (or upload them in your site builder), then submit_urls and crawl again to confirm the check clears. On a WordPress site paired with the AuditAE plugin (0.35.0+), wp_optimize_media does it in place instead: each Media Library file keeps its name and URL, its thumbnail sizes are compressed too and the originals are kept for wp_restore_media. On an unpaired WordPress site a Media Library upload gets a new file name, so the files keep their type and name; replace each one with a media-replace plugin that keeps the URL.
compare_crawls compares the latest crawl with the one before it, or any two by run_id and base_run_id: Site Health and theme changes, new and fixed issues per check, pages added or removed, robots.txt changes and the last 12 crawls' history. Free. get_crawl_issue with compare set to new or fixed lists the URLs behind one check's change, and get_latest_crawl carries the change in brief as vs_previous.
schedule_crawl turns weekly or monthly re-crawls on, changes them or stops them (cadence off); without a cadence, or with dry_run, it only quotes the most a run can cost and the monthly estimate. Free to set; each run bills 1¢ per 10 pages when it runs. get_latest_crawl shows the site's schedule.
REST: GET /api/v1/crawl/{id} returns the same fields plus the stored comparison, GET /api/v1/crawl/{id}/checks lists every check with its result, GET /api/v1/crawl/{id}/checks/{check_id} matches get_crawl_issue (including ?compare=), GET /api/v1/crawl/{id}/compare matches compare_crawls and GET /api/v1/crawl/history?site_id= lists a site's crawls, GET and DELETE /api/v1/crawl/schedule?site_id= read and stop the schedule, and PUT /api/v1/crawl/schedule with { site_id, cadence } in the body sets it. See REST API and API keys.

Check ids never change, so an agent can track a check across crawls. Crawls run before Site Audit checks existed keep their original issue list and say so; crawl again for Site Health.

Deprecated: the older issues_summary and top_problem_pages fields stay in crawl answers for agents that read them. On new crawls the MCP tools return issues_summary as counts only and top_problem_pages as the 5 worst pages with their failing checks; the REST API keeps the lists, each capped at 500 URLs. Read the checks instead; the old fields will be removed no earlier than December 1, 2026.

Every check

173 checks, grouped by their main theme. "Also in" lists the other themes a check counts toward; "Semrush" is the matching Semrush Site Audit issue id.

Crawlability

54 checks. Weighted share of crawlability checks passing (errors 4, warnings 1, notices ¼).

Check	Title	Severity	Also in	Rule	Semrush
`http_5xx`	Pages returning 5XX	Error	—	Final HTTP status 500–599.	1
`http_4xx`	Pages returning 4XX	Error	—	Final HTTP status 400–499, except 401, 403, 407 and 429 (those are http_access_denied).	2
`http_access_denied`	Pages refused the crawler	Warning	AI Search	Status 401, 403, 407, 429 or 999: usually a firewall, bot protection or a login, so kept out of 4XX.	2
`page_unreachable`	Pages couldn't be crawled	Error	—	fetch_error is timeout, dns, conn_refused, conn_reset, tls:*, body_error, bad_redirect_location, unsafe_url or unknown.	9
`dns_failed`	DNS resolution issue	Error	—	The homepage fetch failed with fetch_error dns: the host didn't resolve.	10
`redirect_chain`	Redirect chains and loops	Error	Site Performance	3 or more redirect hops, or fetch_error redirect_loop or redirect_too_many (more than 5 hops).	33
`redirect_temporary`	Temporary redirects	Warning	—	A redirect hop answered 302, 303 or 307.	109
`redirect_permanent`	Permanent redirects	Notice	—	A redirect hop answered 301 or 308, with fewer than 3 hops.	214
`redirect_offsite`	Internal URLs redirecting off-site	Notice	—	An internal URL redirected to another host (fetch_error redirect_offsite).	—
`meta_refresh`	Meta refresh used	Error	—	The page has a `<meta http-equiv="refresh">`.	40
`blocked_by_robots`	Linked URLs blocked by robots.txt	Notice	—	An internal URL linked from crawled pages that robots.txt disallows for the crawl's user agent; of = pages crawled + blocked URLs.	4
`noindex_page`	Noindex pages	Notice	—	Meta robots (or X-Robots-Tag) says noindex or none.	4
`noindex_header`	X-Robots-Tag noindex	Notice	—	The X-Robots-Tag response header says noindex or none.	209
`nofollow_page`	Page-level nofollow	Notice	Internal Linking	Meta robots or X-Robots-Tag says nofollow (or none).	—
`robots_txt_missing`	robots.txt not found	Notice	—	/robots.txt returns 404 or 410.	203
`robots_txt_invalid`	Invalid robots.txt format	Error	—	/robots.txt answers 200 with an HTML page, or with a non-empty body that has no recognised directive line (User-agent, Allow, Disallow, Sitemap, Crawl-delay …).	16
`robots_txt_unknown_lines`	robots.txt has lines crawlers ignore	Notice	—	At least one non-comment line that isn't a known key: value, or an Allow/Disallow before the first User-agent.	16
`robots_txt_unreachable`	robots.txt returns a server error	Error	—	/robots.txt answers 5XX, or the request fails or times out.	—
`robots_txt_too_large`	robots.txt over 500 KiB	Warning	—	robots.txt is larger than 512,000 bytes (Google reads only the first 500 KiB).	—
`search_bots_blocked`	Googlebot or Bingbot blocked	Error	—	robots.txt blocks Googlebot or Bingbot from the whole site (the group that applies to it disallows /).	—
`sitemap_missing`	sitemap.xml not found	Warning	—	No Sitemap: line in robots.txt, and neither /sitemap.xml nor /sitemap_index.xml returns an XML sitemap.	125
`sitemap_not_in_robots`	Sitemap not referenced in robots.txt	Warning	—	robots.txt returns 200 with no Sitemap: line, while a sitemap was found at a default path.	124
`sitemap_invalid`	Invalid sitemap format	Error	—	A sitemap file that answers 200 but is HTML or has no `<urlset>`/`<sitemapindex>` root. Not run when a file couldn't be read (gzip failure, size cap).	17
`sitemap_too_large`	Sitemap file too large	Error	—	A sitemap file with more than 50,000 `<loc>` entries or more than 50 MB uncompressed.	43
`sitemap_bad_urls`	Sitemap URLs that redirect or fail	Error	—	A crawled URL listed in the sitemap whose status isn't 2XX, or that redirected (judged on the `<loc>` form).	18
`sitemap_noindex`	Noindex pages in sitemap	Error	—	A URL listed in the sitemap is noindex.	18
`sitemap_non_canonical`	Non-canonical pages in sitemap	Error	—	A URL listed in the sitemap has a canonical pointing to another URL.	18
`sitemap_blocked`	Sitemap lists robots-blocked URLs	Warning	—	A same-site `<loc>` is disallowed by robots.txt for the crawl's user agent.	—
`sitemap_offsite_urls`	Sitemap lists other hosts	Warning	—	A `<loc>` whose host isn't this site or its www/non-www twin.	—
`indexable_not_in_sitemap`	Indexable pages missing from sitemap	Notice	—	An indexable 200 page that isn't in the sitemap. Not run with no sitemap, a truncated sitemap read, a capped crawl or a scoped crawl.	—
`orphan_sitemap_page`	Orphaned sitemap pages	Notice	Internal Linking	A page in the sitemap that no crawled page links to (inlinks 0), the homepage excepted. Not run on a capped or scoped crawl.	207
`deep_page`	Crawl depth over 3 clicks	Notice	Internal Linking	Reached in 4 or more clicks from the homepage (sitemap-only pages, depth 99, excluded).	212
`url_too_long`	URL over 200 characters	Notice	—	The URL is longer than 200 characters.	201
`url_underscores`	Underscores in URL	Warning	—	The URL path contains an underscore.	122
`url_too_many_params`	Too many URL parameters	Warning	—	More than 2 query parameters.	113
`url_uppercase`	Uppercase letters in URL	Notice	—	The URL path contains uppercase letters.	—
`url_double_slash`	Double slash in URL path	Notice	—	The URL path contains //.	—
`url_session_id`	Session ID in URL	Warning	—	A query parameter named sid, sessionid, session_id, phpsessid, jsessionid, aspsessionid…, or sessid, or a ;jsessionid= path parameter.	—
`canonical_broken`	Broken canonical	Error	—	The canonical target was crawled and returned 4XX/5XX (except 401/403/407/429/999, which mean the crawler was refused) or couldn't be fetched.	38
`canonical_multiple`	Multiple canonical URLs	Error	—	More than one distinct target across all canonical tags and the Link header.	39
`canonical_to_redirect`	Canonical points to a redirect	Warning	—	The canonical target was crawled and redirected.	—
`canonical_to_noindex`	Canonical points to a noindex page	Warning	—	The canonical target was crawled and is noindex.	—
`canonical_elsewhere`	Canonicalised to another URL	Notice	—	The canonical points to a URL other than the page's final URL (and the target isn't broken).	—
`canonical_to_homepage`	Canonical points to the homepage	Warning	—	A page other than the homepage has a canonical pointing at the homepage.	—
`canonical_missing`	No canonical tag	Notice	—	A page without noindex that has no canonical tag or Link header.	—
`canonical_relative`	Relative canonical URL	Notice	—	The canonical href isn't an absolute http(s):// URL.	—
`canonical_outside_head`	Canonical outside head	Warning	—	A canonical `<link>` appears after `</head>` or <body.	—
`malformed_links`	Malformed links	Error	Internal Linking	At least one `<a href>` that fails URL parsing, contains a backslash or whitespace, or is written http:/x or https:x.	44, 11
`link_url_too_long`	Link URLs over 2,000 characters	Warning	—	At least one href longer than 2,000 characters.	136
`soft_404_site`	Missing pages return 200	Warning	—	A request for a random missing path (/auditae-404-`<8 hex>`) returns 2XX.	—
`soft_404_page`	Page looks like an error page	Warning	—	A 2XX page under 300 words whose title or H1 says 404, page not found, not found or page doesn't exist.	—
`mixed_slash_links`	Linked both with and without trailing slash	Notice	Internal Linking	The same page is linked both with and without a trailing slash.	—
`broken_internal_image`	Broken internal images	Error	Site Performance	An internal image that returns 404, 410 or a 5XX other than 503/508/509 (overload or rate limits) or Cloudflare's 520–527, or whose host doesn't exist or refuses connections.	13
`blocked_internal_resources`	Internal resources blocked in robots.txt	Warning	—	An internal image, JS or CSS file disallowed by robots.txt for the crawl's user agent.	130

HTTPS

17 checks. Weighted share of HTTPS checks passing (errors 4, warnings 1, notices ¼).

Check	Title	Severity	Also in	Rule	Semrush
`https_not_used`	Homepage not on HTTPS	Warning	—	The homepage's final URL is http:.	126
`http_not_redirected`	HTTP homepage doesn't redirect to HTTPS	Error	—	http://host/ (redirects not followed) doesn't answer with a 3XX to an https: URL.	32
`http_redirect_temporary`	HTTP→HTTPS redirect is temporary	Warning	—	The first redirect from http:// to https:// is a 302 or 307.	32
`www_not_consolidated`	www and non-www both serve the site	Error	Crawlability	The other host form (www or non-www) answers 2XX without redirecting to the main host and without a canonical to it.	19
`mixed_content`	Mixed content	Error	—	An https page loads at least one http: subresource (image, script, stylesheet, iframe, media).	30
`links_to_http`	Links on HTTPS pages lead to HTTP	Warning	—	At least one `<a href="http:…">` on an https page.	31
`http_password_form`	Password field on an HTTP page	Error	—	An http: page has an `<input type="password">`.	26
`tls_cert_expired`	Certificate expired or about to	Error	—	The certificate's valid_to is in the past or less than 7 days away; a certificate with a lifetime of 30 days or less is judged by the share of its lifetime left (less than 1 day).	27
`tls_cert_expiring`	Certificate expires within 30 days	Warning	—	The certificate expires in 7 to 29 days; a certificate with a lifetime of 30 days or less is judged by the share of its lifetime left (1 day up to a third of its lifetime).	27
`tls_name_mismatch`	Certificate name mismatch	Error	—	tls.checkServerIdentity(host, cert) fails: the certificate doesn't cover this host name.	29
`tls_untrusted`	Untrusted certificate chain	Error	—	The TLS handshake reports an authorization error other than expiry or name: self-signed, incomplete chain or unknown CA.	—
`hsts_missing`	No HSTS	Notice	—	The https homepage sends no Strict-Transport-Security header, or a max-age under 31,536,000.	205
`sitemap_http_urls`	HTTP URLs in an HTTPS site's sitemap	Warning	—	The homepage is https and the sitemap lists at least one same-site http: URL.	128
`canonical_http`	Canonical from HTTPS to HTTP	Warning	Crawlability	An https page whose canonical is an http: URL.	—
`redirect_https_to_http`	HTTPS page redirects to HTTP	Error	—	The redirect chain has a hop from https: to http:.	—
`security_headers_missing`	Homepage missing security headers	Notice	—	The homepage lacks X-Content-Type-Options: nosniff, or lacks both X-Frame-Options and a CSP frame-ancestors directive.	—
`http_pages`	Pages served over HTTP on an HTTPS site	Warning	—	The homepage is https but the page's final URL is http:.	—

International SEO

13 checks. Weighted share of hreflang and language checks passing (errors 4, warnings 1, notices ¼).

Check	Title	Severity	Also in	Rule	Semrush
`hreflang_invalid_code`	Invalid hreflang values	Error	—	An hreflang value that isn't x-default or language[-Script][-REGION] (ISO 639-1, ISO 15924, ISO 3166-1 alpha-2 or 419); en-UK is invalid (it's en-GB).	23
`hreflang_no_self`	Hreflang without self-reference	Error	—	The page has hreflang entries but none points at its own URL.	24
`hreflang_conflicting`	Conflicting hreflang URLs	Error	—	One hreflang code maps to two or more different URLs.	24
`hreflang_canonical_conflict`	Hreflang on a canonicalised page	Error	—	The page has hreflang entries and its canonical points to another URL.	24
`hreflang_relative`	Relative hreflang URLs	Error	—	An hreflang href isn't an absolute URL.	25
`hreflang_broken`	Hreflang links to broken pages	Error	—	An hreflang target was crawled and returned 4XX/5XX.	25
`hreflang_redirect`	Hreflang links to redirects	Error	—	An hreflang target was crawled and redirected.	25
`hreflang_no_return`	Missing return hreflang	Warning	—	Page A lists B, B was crawled and parsed, and B's hreflang set has no URL equal to A.	—
`hreflang_no_x_default`	No x-default hreflang	Notice	—	Two or more hreflang entries and no x-default.	—
`no_lang`	No hreflang and no lang attribute	Warning	—	No `<html lang>` and no hreflang entries.	114
`lang_invalid`	Invalid html lang	Warning	—	The `<html lang>` primary subtag isn't an ISO 639-1 or 3-letter language code, or the value isn't a well-formed BCP 47 tag (en_US, for one).	—
`hreflang_lang_mismatch`	Hreflang language differs from html lang	Notice	—	The language of the page's own hreflang entry differs from its `<html lang>` (we compare the declarations; Semrush detects the text's language).	204
`content_language_mismatch`	Content-Language differs from html lang	Notice	—	The Content-Language header's language differs from `<html lang>`'s primary subtag.	—

Core Web Vitals

5 checks. (good pages + ½ needs-improvement pages) ÷ pages measured, PageSpeed mobile on up to 10 pages; never part of Site Health.

Check	Title	Severity	Also in	Rule	Semrush
`cwv_lcp_poor`	Poor Largest Contentful Paint	Warning	—	Lab LCP (PageSpeed, mobile) over 4.0 s on a sampled page (≤10 pages).	—
`cwv_cls_poor`	Poor Cumulative Layout Shift	Warning	—	Lab CLS over 0.25 on a sampled page.	—
`cwv_tbt_poor`	Poor Total Blocking Time	Warning	—	Lab Total Blocking Time over 600 ms on a sampled page (the lab stand-in for INP).	—
`cwv_inp_poor`	Poor Interaction to Next Paint (field)	Warning	—	Field INP from the Chrome UX Report (via PageSpeed) over 500 ms, for the URL or else the origin.	—
`cwv_needs_improvement`	Core Web Vitals need improvement	Notice	—	No metric poor, but LCP 2.5–4 s, CLS 0.1–0.25, TBT 200–600 ms or INP 200–500 ms on a sampled page.	—

Site Performance

17 checks. Weighted share of performance checks passing (errors 4, warnings 1, notices ¼).

Check	Title	Severity	Also in	Rule	Semrush
`html_too_large`	HTML over 2 MB	Error	—	The HTML response reached the 2,000,000-byte read cap.	21
`html_large`	HTML over 1 MB	Warning	—	HTML between 1,000,000 bytes and the 2 MB read cap.	—
`slow_page_load`	Slow page (HTML) load	Error	—	The final hop's response headers plus body took over 3,000 ms (measured from a Vercel region).	111
`slow_ttfb`	Slow server response	Notice	—	Time to first byte on the final hop over 800 ms.	—
`html_uncompressed`	Uncompressed pages	Warning	—	HTML over 1 KiB sent with no Content-Encoding (gzip, br, zstd or deflate) although the request offered gzip and br.	129
`js_css_uncompressed`	Uncompressed JS/CSS	Warning	—	An internal JS or CSS file over 1 KiB served with no Content-Encoding.	131
`js_css_uncached`	Uncached JS/CSS	Warning	—	An internal JS or CSS file with no Cache-Control max-age/s-maxage over 0 or immutable, and no future Expires.	132
`js_css_too_large`	Page JS+CSS over 2 MB	Warning	—	The known sizes of a page's JS and CSS files add up to more than 2 MB; not judged for a page when over half its files have no known size.	133
`js_css_too_many`	Too many JS/CSS files	Warning	—	More than 100 script src plus stylesheet links.	134
`js_css_unminified`	Unminified JS/CSS	Warning	—	An internal file of 2 KB or more, without .min. in its name, whose first 64 KB is over 20% whitespace with lines averaging under 120 characters (≤200 files sampled).	135
`render_blocking_scripts`	Render-blocking scripts	Notice	Core Web Vitals	A `<script src>` in the head without async, defer or type=module (nomodule and non-JavaScript scripts are skipped: browsers never run them).	—
`images_no_dimensions`	Images without width and height	Notice	Core Web Vitals	At least one `<img>` missing a width or height attribute.	—
`large_images`	Images over 200 KB	Notice	—	An internal image whose size (Content-Length) is over 200 KB.	—
`http2_unsupported`	No HTTP/2	Notice	—	The TLS handshake (ALPN) didn't negotiate h2.	—
`broken_external_image`	Broken external images	Warning	—	An external image that returns 404, 410 or a 5XX other than 503/508/509 (overload or rate limits) or Cloudflare's 520–527, or whose host doesn't exist or refuses connections.	14
`broken_internal_js_css`	Broken internal JS/CSS	Error	Crawlability	An internal JS or CSS file that returns 404, 410 or a 5XX other than 503/508/509 (overload or rate limits) or Cloudflare's 520–527, or whose host doesn't exist or refuses connections; every page that loads it is flagged.	41
`broken_external_js_css`	Broken external JS/CSS	Notice	—	An external JS or CSS file that returns 404, 410 or a 5XX other than 503/508/509 (overload or rate limits) or Cloudflare's 520–527, or whose host doesn't exist or refuses connections.	211

Internal Linking

15 checks. Weighted share of internal-linking checks passing (errors 4, warnings 1, notices ¼).

Check	Title	Severity	Also in	Rule	Semrush
`broken_internal_link`	Broken internal links	Error	Crawlability	A crawled internal URL that other pages link to returned 4XX/5XX (except 401/403/407/429/999, which mean the crawler was refused) or couldn't be fetched; n and of count links (Σ inlinks of broken targets ÷ Σ inlinks).	8
`link_to_redirect`	Internal links to redirects	Notice	—	A crawled internal URL that redirected (judged on the linked form) and that other pages link to.	—
`link_to_blocked`	Internal links to robots-blocked pages	Notice	—	An internal URL disallowed by robots.txt that crawled pages link to.	—
`nofollow_internal`	Nofollow internal links	Warning	—	At least one internal `<a>` whose rel includes nofollow.	123
`nofollow_external`	Nofollow external links	Notice	—	At least one external `<a>` whose rel includes nofollow.	202
`too_many_links`	Too many on-page links	Warning	—	More than 3,000 `<a href>` on the page.	108
`single_inlink`	Only one incoming internal link	Notice	AI Search	A page that exactly one internal link points to (the homepage excepted). Not run on a capped or scoped crawl.	213
`nofollow_only_inlinks`	Linked only through nofollow	Notice	—	Every internal link to the page is nofollow.	—
`no_outlinks`	No outgoing internal links	Notice	—	A parsed page with no internal links to other pages.	—
`empty_anchor`	Links with no anchor text	Notice	AI Search	At least one `<a>` with no text, no image alt, no aria-label and no title.	216
`generic_anchor`	Non-descriptive anchor text	Notice	AI Search	Anchor text such as click here, read more, learn more, here or more info, a raw URL, or symbols only.	217
`resource_as_page_link`	Links to files formatted as page links	Notice	—	An `<a href>` to an image, script, stylesheet or document file (pdf, docx, zip …).	215
`javascript_links`	javascript: links	Notice	AI Search	At least one href="javascript:…".	—
`broken_external_link`	Broken external links	Warning	—	An external link that returns 404, 410 or a 5XX other than 503/508/509 (overload or rate limits) or Cloudflare's 520–527, or whose host doesn't exist or refuses connections.	12
`external_link_403`	External links returning 403	Notice	—	An external link that answered 403 (often bot protection; counted as unverified, never broken).	218

Markup

8 checks. Valid structured-data items ÷ all items (Semrush's rule); no score when the site has none.

Check	Title	Severity	Also in	Rule	Semrush
`schema_invalid`	Invalid structured data items	Error	—	validate_schema reports at least one error on a JSON-LD node of a type it has rules for (≤10 blocks, ≤100 KB a block and ≤30 nodes a page are checked).	45
`schema_parse_error`	Unparseable JSON-LD	Error	—	JSON.parse fails on an application/ld+json block.	—
`schema_warnings`	Structured data with warnings	Notice	—	At least one validation warning, or an error on a type without rules, and no counted errors.	—
`schema_missing`	No structured data	Notice	AI Search	An indexable page with no JSON-LD and no microdata or RDFa.	—
`schema_org_missing`	No Organization or WebSite markup on the homepage	Notice	AI Search	The homepage's schema types include no Organization (or a subtype such as LocalBusiness) and no WebSite.	—
`og_missing`	Open Graph tags missing	Notice	—	An indexable page lacking og:title or og:image.	—
`twitter_card_missing`	Twitter card missing	Notice	—	An indexable page with no twitter:card meta tag.	—
`amp_no_canonical`	AMP page with no canonical	Error	—	An `<html amp>` or `<html ⚡>` page without a canonical.	22

AI Search

18 checks. Weighted share of AI-search checks passing (errors 4, warnings 1, notices ¼).

Check	Title	Severity	Also in	Rule	Semrush
`llms_txt_missing`	llms.txt not found	Notice	—	/llms.txt isn't found.	137
`llms_txt_invalid`	llms.txt formatting issues	Notice	—	/llms.txt answers 200 but has a problem or formatting errors.	219
`content_too_long`	Too much content	Notice	—	More than 10,000 words.	220
`outdated_content`	Outdated content	Notice	—	The Last-Modified header is more than 183 days old; pages without the header aren't counted.	221
`low_semantic_html`	Low semantic HTML	Notice	—	Semantic tags (main, article, section, nav, header, footer, aside, figure, h1–h6, p, lists, tables …) are under 15% of semantic + div + span tags.	222
`heading_hierarchy`	Skipped heading level	Notice	On-page and content	A heading level is skipped (e.g. an H4 straight after an H2).	223
`long_paragraphs`	Paragraphs over 150 words	Notice	—	At least one `<p>` over 150 words.	223
`low_readability`	Hard-to-read text	Notice	—	Flesch Reading Ease under 30, on English (or undeclared) pages of 200+ words.	223
`ai_search_bots_blocked`	AI search bots blocked in robots.txt	Error	—	robots.txt blocks OAI-SearchBot, Claude-SearchBot or PerplexityBot from the whole site (Semrush KB 1571).	—
`ai_user_fetchers_blocked`	AI user fetchers blocked	Warning	—	robots.txt blocks ChatGPT-User, Claude-User or Perplexity-User from the whole site.	—
`page_blocked_for_ai`	Pages blocked from AI search	Notice	—	A crawled page that robots.txt disallows for OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Perplexity-User, Googlebot or Google-Extended.	—
`ai_training_bots_blocked`	AI training bots blocked	Notice	—	robots.txt blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot or Meta-ExternalAgent site-wide (a policy choice; informational).	—
`ai_fetcher_firewall`	AI fetchers blocked by the firewall	Warning	—	robots.txt allows ChatGPT-User and AuditAE-Bot got 2XX, but the homepage fetched with an AI agent's user-agent is refused (401/403/429/503…) or challenged.	—
`content_signal_restrictive`	Content-Signal limits AI use	Notice	—	robots.txt's Content-Signal sets search=no or ai-input=no.	—
`nosnippet`	Snippets blocked	Warning	—	An indexable page with nosnippet or max-snippet:0 for all crawlers.	—
`js_dependent_content`	Content needs JavaScript	Warning	—	A 2XX page with under 50 words, at least one script file and visible text under 2% of the HTML.	—
`article_no_date`	Articles without a date	Notice	—	An article-like page (og:type article with an article:published_time/author/section/tag, or Article/BlogPosting/NewsArticle markup) with no datePublished, dateModified or article:*_time.	—
`article_no_author`	Articles without an author	Notice	Markup	An article-like page with no JSON-LD author, meta author or rel=author.	—

On-page and content (AuditAE extra)

26 checks. Weighted share of title, description, heading and content checks passing (errors 4, warnings 1, notices ¼). An AuditAE extra; Semrush has no such theme.

Check	Title	Severity	Also in	Rule	Semrush
`title_missing`	Title tag missing or empty	Error	—	No `<title>`, or one with only whitespace.	3
`title_duplicate`	Duplicate title tags	Error	—	The same title (trimmed, whitespace collapsed, case-insensitive) on 2 or more indexable pages.	6
`title_too_short`	Title too short	Warning	—	A title of 1–10 characters.	101
`title_too_long`	Title too long	Warning	—	A title over 70 characters.	102
`title_multiple`	Multiple title tags	Warning	—	More than one `<title>` in the head (SVG titles in the body are ignored).	—
`meta_description_missing`	Missing meta description	Warning	—	No meta description, or an empty one.	106
`meta_description_duplicate`	Duplicate meta descriptions	Error	—	The same normalised meta description on 2 or more indexable pages.	15
`meta_description_multiple`	Multiple meta descriptions	Warning	—	More than one meta name="description" tag.	—
`meta_description_length`	Meta description too short or long	Notice	—	A meta description under 70 or over 160 characters.	—
`h1_missing`	Missing H1	Warning	—	No `<h1>` on the page.	103
`h1_multiple`	More than one H1	Notice	—	More than one `<h1>`.	104
`h1_same_as_title`	H1 duplicates the title	Warning	—	The first H1 equals the title (trimmed, whitespace collapsed, case-insensitive).	105
`h1_empty`	Empty H1	Notice	—	At least one `<h1>` with no text and no image alt.	—
`h1_duplicate`	Same H1 on several pages	Notice	—	The same normalised first H1 on 2 or more indexable pages.	—
`images_missing_alt`	Missing ALT attributes	Warning	—	At least one `<img>` with no alt attribute (alt="" passes, an unquoted alt counts); of = pages with images.	110
`low_word_count`	Low word count	Warning	—	An indexable page with fewer than 200 words.	117
`low_text_html_ratio`	Low text to HTML ratio	Warning	—	Visible text is 10% or less of the HTML.	112
`duplicate_content`	Duplicate content	Error	—	Indexable pages with 100+ words whose main-text hash is equal, or whose SimHash is ≤6 bits apart, unless one canonicalises to the other; groups say exact or near.	7
`no_viewport`	No viewport tag	Error	—	No `<meta name="viewport">`.	20
`viewport_no_width`	Viewport width not set	Error	—	A viewport tag without width= or initial-scale=.	46
`doctype_missing`	Doctype not declared	Warning	—	No `<!doctype html>` in the first 2 KB (after a byte-order mark and comments).	116
`charset_missing`	Encoding not declared	Warning	—	No charset in the Content-Type header, a `<meta charset>` or an http-equiv Content-Type.	115
`frames_used`	Frames used	Warning	—	The page uses `<frameset>` or `<frame>`.	121
`incompatible_plugins`	Flash, Java or Silverlight	Warning	—	An `<applet>`, or an `<object>`/`<embed>` with Flash, Java or Silverlight.	120
`lorem_ipsum`	Placeholder text	Warning	—	"lorem ipsum" in the main text.	—
`deprecated_html`	Deprecated HTML tags	Notice	—	`<center>`, `<font>`, `<marquee>` or `<blink>` tags.	—

Try it in AuditAE

Crawl a site

Still stuck? Signed in, ask AEBOT in Documentation mode (free). Or email support@auditae.app.