Does AhrefsSiteAudit respect robots.txt?

Yes by default, according to Ahrefs. A verified site owner can deliberately authorize the crawler to ignore robots.txt on that site, so compliance is conditional when that option is enabled.

Why did AhrefsSiteAudit send many requests together?

An audit works through the pages in its configured scope and can create a noticeable batch of traffic. The project owner can reduce the crawl rate or reschedule the run.

Is AhrefsSiteAudit the same as AhrefsBot?

No. AhrefsSiteAudit checks a site for a user's audit project. AhrefsBot maintains Ahrefs' wider link database and supplies the Yep search index.

Does this crawler send pages to AI training?

No training purpose is documented. Its output is a Site Audit report for the Ahrefs user who configured the crawl.

What happens if I block AhrefsSiteAudit?

The audit may omit blocked pages and miss issues there. Blocking this token does not remove the site from AhrefsBot's database or from any unrelated AI service.

AhrefsSiteAudit & AI Search: What to Know

What is AhrefsSiteAudit?

AhrefsSiteAudit appears when an Ahrefs user runs Site Audit against a website. The crawler loads and renders pages to find technical and on-page SEO problems, including broken links, performance issues, and security misconfigurations. It is tied to an audit project rather than Ahrefs' continuous crawl of the wider web.

The crawler reports those issues to the Ahrefs user who configured the project. It does not build a public AI answer index, and Ahrefs does not identify it as collecting foundation-model training data. Fixes prompted by an audit may make a site easier for many crawlers to access, but an AhrefsSiteAudit visit itself has no documented AI-search effect.

Requests tend to arrive as the configured audit works through its URL set. A site owner may initiate the run manually or schedule it in Ahrefs, so traffic can be concentrated around audit windows. Cloudflare's bot directory records Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/) as an example header.

Ahrefs says both of its bots obey robots.txt and crawl-delay directives by default. Site Audit has one important exception: a verified site owner can authorize AhrefsSiteAudit to ignore robots.txt for that site so the audit can inspect sections normally hidden from crawlers. Cloudflare's bot directory marks compliance false, which does not capture this default-plus-override behavior.

Indirectly relevant

Is AhrefsSiteAudit relevant for AI search?

Indirectly. AhrefsSiteAudit has no AI product of its own, but its output can end up in the systems that AI answers draw on.

AhrefsSiteAudit is an SEO auditing crawler. It has no direct effect on search rankings or AI answers, but the reports it feeds are used to fix crawlability and content issues that do affect how search and AI systems see your site.

How to handle AhrefsSiteAudit

If your team is running the audit, allow the intended public scope and tune crawl speed in Ahrefs or with a supported crawl delay. A site that does not want this crawler can publish:

User-agent: AhrefsSiteAudit
Disallow: /

The bot follows robots.txt by default, but a verified site owner can enable Ahrefs' documented override. If it reaches a disallowed path, first check the Site Audit project and who verified the domain. Authentication remains necessary for any page that should not be exposed to an authorized audit.

Examples

A retailer schedules a weekly Site Audit, then sees AhrefsSiteAudit fetch product and category pages during the configured run.
A verified owner authorizes an audit of a staging section that ordinary crawlers cannot reach through robots.txt.
An operations team traces a morning request spike to an Ahrefs audit and lowers its crawl speed instead of blocking the tool.

Frequently asked questions about AhrefsSiteAudit

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Ahrefs operates it for the Site Audit tool. Ahrefs customers use the crawler to inspect websites for technical and on-page SEO issues.

Related bots

AhrefsBot

Powers the database for both Ahrefs, a marketing intelligence platform, and Yep, an independent, privacy-focused search engine.

SEO

Barkrowler

Barkrowler is Babbar's web crawler that fuels and updates their graph representation of the web, providing SEO tools for the marketing community.

SEO

ClarityBot

ClarityBot is seoClarity's web crawler that performs technical SEO audits, analyzes content, and monitors website performance.

SEO

DataForSeoBot

DataForSeoBot is a backlink checker bot operated by DataForSEO that crawls websites to build and maintain their backlink database.

SEO

DotBot

DotBot is a web crawler operated by Moz (formerly SEOmoz) that collects data for their Link Explorer tool and Links API.

SEO

AhrefsSiteAudit