Why does it revisit a 301 or 404 URL?

Serpstat keeps historical link data. While other pages still link to that URL, the crawler can continue finding and checking it.

Does SerpstatBot respect robots.txt?

Serpstat says it does, although this bot does not independently confirm compliance. A failed robots.txt request is treated as no restriction, so the file must remain reachable.

Can I slow SerpstatBot without blocking it?

The operator supports an integer `Crawl-delay` of up to 20 seconds. Place it in the `serpstatbot` group and verify the resulting request rate.

Does SerpstatBot crawl for AI search or training?

Serpstat identifies a backlink index as the output. It does not assign the bot an AI search or model-training purpose.

SerpstatBot & AI Search: What to Know

What is SerpstatBot?

SerpstatBot crawls the web to build Serpstat's backlink index. It follows links to discover new ones and returns to track changes, making its data available through Serpstat's Backlink Analysis product. A visit is part of that web-wide collection process, not a site audit requested by the publisher.

The bot scans links marked rel="nofollow". It also retains historical link information, so it can keep revisiting a URL that returns 301 or 404 while links to that URL still exist. Those repeated requests may be intentional index maintenance rather than a failure to notice the response status.

Serpstat's bot documentation says the crawler follows robots.txt and supports Allow, Disallow, wildcard matching, and integer Crawl-delay values up to 20 seconds. If it cannot retrieve robots.txt, including when the server returns 403, it treats the site as having no restrictions. The structured record here leaves compliance unconfirmed, so the operator's stated behavior should still be tested against access logs.

The fetched link data supports an SEO backlink database. Serpstat does not describe this crawler as retrieving sources for generative answers or collecting page text for model training. A SerpstatBot visit may change what Serpstat reports about links, but it has no documented direct effect on AI citations.

Indirectly relevant

Is SerpstatBot relevant for AI search?

Indirectly. SerpstatBot has no AI product of its own, but its output can end up in the systems that AI answers draw on.

SerpstatBot is an SEO auditing crawler. It has no direct effect on search rankings or AI answers, but the reports it feeds are used to fix crawlability and content issues that do affect how search and AI systems see your site.

How to handle SerpstatBot

Allow SerpstatBot if you want its crawler to observe the site's links for Serpstat's backlink index. To stop the documented user agent, Serpstat provides this rule:

User-agent: serpstatbot
Disallow: /

Make sure /robots.txt itself returns a readable file. Serpstat says a 403 or another retrieval failure is treated as permission to crawl. It follows redirects to the file only within the same site.

If the crawl is useful but too frequent, add an integer Crawl-delay no greater than 20 under the serpstatbot group. The operator says longer values are reduced to 20. Since compliance is not confirmed by the structured record, watch the request cadence after the change.

Use an origin or edge block for a firm refusal. The record contains several known user-agent examples but no verified IP range or signature directory, so matching the string does not prove who sent a request.

Examples

A backlink analyst leaves SerpstatBot available so new links from the site can enter Serpstat's index.
A removed campaign page keeps receiving visits because external pages still link to its old URL and the bot preserves link history.
An administrator fixes a 403 response on `/robots.txt`, then confirms that the crawler begins honoring the site's disallow rule.

Frequently asked questions about SerpstatBot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Serpstat adds discovered links and later changes to its backlink index, which customers access through Backlink Analysis.

Related bots

AhrefsBot

Powers the database for both Ahrefs, a marketing intelligence platform, and Yep, an independent, privacy-focused search engine.

SEO

AhrefsSiteAudit

Powers Ahrefs’ Site Audit tool. Ahrefs users can use Site Audit to analyze websites and find both technical SEO and on-page SEO issues.

SEO

Barkrowler

Barkrowler is Babbar's web crawler that fuels and updates their graph representation of the web, providing SEO tools for the marketing community.

SEO

ClarityBot

ClarityBot is seoClarity's web crawler that performs technical SEO audits, analyzes content, and monitors website performance.

SEO

DataForSeoBot

DataForSeoBot is a backlink checker bot operated by DataForSEO that crawls websites to build and maintain their backlink database.

SEO

SerpstatBot