Does AhrefsBot respect robots.txt?

Ahrefs says yes and documents support for `Allow`, `Disallow`, and `Crawl-delay`. The metadata also says yes, while Cloudflare's bot directory field conflicts, so confirm unusual behavior in your own logs.

Why does AhrefsBot revisit old or broken URLs?

Its index needs current link and status information. Rechecking a known URL lets Ahrefs detect redirects, removals, or pages that have returned.

How is AhrefsBot different from AhrefsSiteAudit?

AhrefsBot crawls the wider web for Ahrefs and Yep. AhrefsSiteAudit runs in connection with a user's site-audit project and looks for technical or on-page issues.

Does AhrefsBot improve AI search visibility?

No direct AI-search or training role is documented. It affects Ahrefs data and Yep's conventional search index, not eligibility for unrelated generated answers.

Can I verify AhrefsBot from the header?

The `AhrefsBot` token and robot URL help classify a request, but this bot has no verified IP status or Web Bot Auth directory. Do not treat the user agent as authentication.

AhrefsBot & AI Search: What to Know

What is AhrefsBot?

AhrefsBot is Ahrefs' broad web crawler. It discovers links, fetches pages, and revisits known URLs to keep Ahrefs' marketing-intelligence database current. The same crawl data also supports Yep, Ahrefs' independent search engine.

This is ongoing index maintenance rather than a site audit requested for one domain. Logs may show AhrefsBot following newly found links or checking the status of pages it has seen before. Cloudflare's bot directory records the current example identity as Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/), with AhrefsBot as the stable token.

Ahrefs' bot documentation says AhrefsBot obeys Allow, Disallow, and Crawl-delay rules. The bot metadata agrees, although Cloudflare's followsRobotsTxt field says false. Since the direct operator documentation is explicit, use its token and review logs if a new rule appears not to have been picked up yet.

AhrefsBot is not documented as a model-training crawler or an AI answer fetcher. Allowing it may affect backlink reports, Ahrefs metrics, and a page's inclusion in Yep. Any benefit to AI visibility would be speculative, so its traffic should remain classified as SEO and conventional search crawling.

Indirectly relevant

Is AhrefsBot relevant for AI search?

Indirectly. AhrefsBot has no AI product of its own, but its output can end up in the systems that AI answers draw on.

AhrefsBot is an SEO auditing crawler. It has no direct effect on search rankings or AI answers, but the reports it feeds are used to fix crawlability and content issues that do affect how search and AI systems see your site.

How to handle AhrefsBot

Keep AhrefsBot available when you want Ahrefs to discover backlinks to the site or want pages represented in Yep. If load is the concern, Ahrefs supports Crawl-delay; if no Ahrefs or Yep coverage is wanted, block the stable token:

User-agent: AhrefsBot
Disallow: /

Ahrefs notes that the crawler may not collect a robots.txt change until its next scheduled crawl. Check later requests before deciding that a new rule was ignored. Use authentication for private content regardless of the rule.

Examples

An SEO analyst leaves AhrefsBot allowed so newly earned links can be reflected in the site's Ahrefs backlink reports.
An infrastructure team adds a crawl delay after repeated AhrefsBot fetches compete with a resource-heavy publishing job.
A publisher blocks AhrefsBot on a licensed archive while keeping public article URLs available to Yep and Ahrefs.

Frequently asked questions about AhrefsBot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Ahrefs operates AhrefsBot. It supplies crawl data to the Ahrefs marketing-intelligence platform and the Yep search engine.

Related bots

AhrefsSiteAudit

Powers Ahrefs’ Site Audit tool. Ahrefs users can use Site Audit to analyze websites and find both technical SEO and on-page SEO issues.

SEO

Barkrowler

Barkrowler is Babbar's web crawler that fuels and updates their graph representation of the web, providing SEO tools for the marketing community.

SEO

ClarityBot

ClarityBot is seoClarity's web crawler that performs technical SEO audits, analyzes content, and monitors website performance.

SEO

DataForSeoBot

DataForSeoBot is a backlink checker bot operated by DataForSEO that crawls websites to build and maintain their backlink database.

SEO

DotBot

DotBot is a web crawler operated by Moz (formerly SEOmoz) that collects data for their Link Explorer tool and Links API.

SEO

AhrefsBot