Does DotBot respect robots.txt?

Yes according to Moz's crawler guide and this bot. Cloudflare's false flag conflicts with that token-specific operator statement.

Why might DotBot continue after a robots.txt change?

Moz says DotBot reads robots.txt when it first encounters the site during a new index crawl. It may not notice a later change until another crawl discovers links to the site.

How is DotBot different from Rogerbot?

DotBot builds Moz's shared link index. Rogerbot performs site audits for Moz Pro Campaigns and On-Demand Crawl.

Will blocking DotBot change search rankings or AI answers?

Blocking it can remove data from Moz products and affect Moz's view of a backlink profile. It does not directly change rankings in an unrelated search engine, and no generative answer or model-training use is documented.

Can DotBot be authenticated?

Not with the supplied data. There is no verified IP range or signed-agent directory, so a matching `DotBot` string identifies a claim rather than proving Moz sent it.

DotBot & AI Search: What to Know

What is DotBot?

DotBot crawls the web for Moz's Link Index. The resulting link data appears in Link Explorer, the Links section of Moz Pro Campaigns, and the Moz Links API.

Its job is to discover pages and the links between them. Moz uses that index for backlink research and metrics including Domain Authority and Page Authority. If DotBot cannot read a page, links found only on that page may be absent from Moz's data even though the page remains available to search engines and visitors.

DotBot is easy to confuse with Rogerbot because both are operated by Moz. Rogerbot audits a site selected in Moz Pro and returns site health findings to that customer. DotBot ranges across the web to maintain a shared link index, so its requests are not evidence that someone ordered an audit of the site.

Moz's Dotbot guide says the crawler always respects the standard Robots Exclusion Protocol and accepts a crawl delay. Cloudflare's directory flag says otherwise. The operator's token-specific documentation and this entry both record robots support as true, which is more specific than the Cloudflare flag.

Robots changes may not stop the traffic immediately. Moz says DotBot reads the file the first time it encounters a site during a new index crawl, then checks again when a later index crawl discovers links to that site. Site owners should allow for that refresh cycle before concluding that a new rule was ignored.

Cloudflare's full example contains DotBot/1.1, an old Open Site Explorer URL, and a Moz help address. No signature key directory or verified IP range is supplied, so this page remains marked unverifiable even though Moz publicly documents the crawler. DotBot has no stated generative answer or model-training role, and its index does not control rankings in Google, Yahoo, or Yandex.

Indirectly relevant

Is DotBot relevant for AI search?

Indirectly. DotBot has no AI product of its own, but its output can end up in the systems that AI answers draw on.

DotBot is an SEO auditing crawler. It has no direct effect on search rankings or AI answers, but the reports it feeds are used to fix crawlability and content issues that do affect how search and AI systems see your site.

How to handle DotBot

Keep DotBot allowed if you want Moz products to observe links on and to the site. Block it if Moz's link index is not useful to you, or disallow expensive sections that contribute little link information.

Moz documents the lowercase product name in robots.txt; the directory token uses different casing:

User-agent: dotbot
Disallow: /

You can replace the root disallow with selected paths. Moz also supports Crawl-delay when the goal is to reduce request frequency instead of leaving its index.

After a policy change, check more than one crawl cycle because DotBot does not re-read robots.txt on every request. If traffic never follows the rule, treat the claimant as unverified. Neither the user agent nor the supplied Cloudflare entry authenticates the sender.

Examples

A publisher keeps DotBot allowed so links inside its public articles can be represented in Moz's Link Index.
An application disallows `/reports/export/` for DotBot but leaves ordinary documentation pages crawlable for backlink analysis.
An administrator updates robots.txt and waits for a later Moz index crawl before investigating a small amount of continued DotBot traffic.

Frequently asked questions about DotBot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

It gathers web and link data for Moz's Link Index. Moz exposes that data through Link Explorer, the Links section of a Campaign, and the Links API.

Related bots

AhrefsBot

Powers the database for both Ahrefs, a marketing intelligence platform, and Yep, an independent, privacy-focused search engine.

SEO

AhrefsSiteAudit

Powers Ahrefs’ Site Audit tool. Ahrefs users can use Site Audit to analyze websites and find both technical SEO and on-page SEO issues.

SEO

Barkrowler

Barkrowler is Babbar's web crawler that fuels and updates their graph representation of the web, providing SEO tools for the marketing community.

SEO

ClarityBot

ClarityBot is seoClarity's web crawler that performs technical SEO audits, analyzes content, and monitors website performance.

SEO

DataForSeoBot

DataForSeoBot is a backlink checker bot operated by DataForSEO that crawls websites to build and maintain their backlink database.

SEO

DotBot