Is `ScreamingFrogBot` the official default user agent?

The directory stores that token, but Screaming Frog documents `Screaming Frog SEO Spider` as the default. The software can send several preset or custom identities.

Does Screaming Frog respect robots.txt?

Normal Spider mode respects it by default. A user can select an ignore option, and List mode is configured to ignore robots.txt by default, so the file is not an access-control boundary.

Does the crawler render JavaScript?

It can. Raw HTML is the default, while a user can enable JavaScript rendering through headless Chromium when the audit needs rendered content or links.

Does Screaming Frog affect AI search or training?

No direct role is documented. Its requests collect data for the user's technical SEO audit rather than a public AI answer index or training pipeline.

Can a user-agent string verify a Screaming Frog request?

No. this bot is marked unverifiable, and the tool itself permits custom user agents. Correlate traffic with an expected audit or enforce normal access rules.

ScreamingFrogBot & AI Search: What to Know

What is ScreamingFrogBot?

Screaming Frog SEO Spider is software that a person runs to audit a chosen website. In its normal Spider mode, it starts from a URL and discovers pages through links. It can crawl raw HTML or use headless Chromium to render JavaScript, while List mode checks URLs supplied by the user.

The crawl populates the user's local audit with response codes, links, directives, page elements, and other technical SEO data. It does not create a public search index. Screaming Frog assigns these fetches no model-training or generative-answer function.

This directory entry uses the token ScreamingFrogBot, but Screaming Frog's own documentation names the default user agent Screaming Frog SEO Spider. The software can also send preset browser and search-engine user agents or a custom string. Because this entry is marked unverifiable and has no matching Cloudflare record, a request that merely claims ScreamingFrogBot should not be attributed to the vendor with confidence.

The SEO Spider respects robots.txt by default and lets the user inspect blocked URLs. A user can deliberately switch to an ignore setting, and List mode starts with robots.txt ignored by default. The recorded true value describes normal Spider behavior, not a security guarantee against a person who configures the tool to bypass the file.

Indirectly relevant

Is ScreamingFrogBot relevant for AI search?

Indirectly. ScreamingFrogBot has no AI product of its own, but its output can end up in the systems that AI answers draw on.

ScreamingFrogBot is an SEO auditing crawler. It has no direct effect on search rankings or AI answers, but the reports it feeds are used to fix crawlability and content issues that do affect how search and AI systems see your site.

How to handle ScreamingFrogBot

If your team or an agency is running the audit, agree on the target, crawl speed, and request identity before it starts. That makes the traffic easier to recognize and prevents an audit from competing with production load.

To cover the official default identity and the token stored in this directory, use separate groups:

User-agent: Screaming Frog SEO Spider
Disallow: /

User-agent: ScreamingFrogBot
Disallow: /

The official default respects robots.txt, but the operator gives users an explicit ignore option. Use authentication, a firewall, or an origin rule when blocked content must remain inaccessible. Do not allowlist either string by itself because the software supports custom identities and any HTTP client can copy a header.

Examples

An SEO consultant starts a Spider-mode crawl after a release and the server logs the official `Screaming Frog SEO Spider` identity.
A developer uploads a migration URL list, then confirms the List-mode settings before requesting pages that the live robots.txt disallows.
A security analyst sees `ScreamingFrogBot`, notes that it is not the vendor's documented default token, and treats the sender as unverified.

Frequently asked questions about ScreamingFrogBot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

A person launches the SEO Spider against a website or uploads a list of URLs. It is user-operated audit software, not an autonomous public indexer.

Related bots

Lumar

Lumar runs cloud-based crawls for sites selected by its customers and turns the fetched page data into private website analytics.

SEO

Sitebulb

Sitebulb is a desktop and cloud-based website crawler used by SEO professionals for technical SEO audits.

SEO

AhrefsBot

Powers the database for both Ahrefs, a marketing intelligence platform, and Yep, an independent, privacy-focused search engine.

SEO

AhrefsSiteAudit

Powers Ahrefs’ Site Audit tool. Ahrefs users can use Site Audit to analyze websites and find both technical SEO and on-page SEO issues.

SEO

Barkrowler

Barkrowler is Babbar's web crawler that fuels and updates their graph representation of the web, providing SEO tools for the marketing community.

SEO

ScreamingFrogBot