How does an OnCrawl crawl begin?

A customer configures a crawl profile with one or more start URLs and can limit discovery with URL rules or subdomain settings.

What is Oncrawl's virtual robots.txt?

It is a crawl-specific policy configured inside Oncrawl. A verified site owner can use it to override the site's published robots file for that project.

Does OnCrawl respect the site's robots.txt?

The local metadata is inconclusive, while Cloudflare says it does not. Oncrawl also documents an authorized virtual override for verified domains, so use an enforceable control for hard restrictions.

Which OnCrawl token belongs in robots.txt?

Oncrawl's own examples use `OnCrawl`. The `www.oncrawl.com` value in this bot appears inside the recorded full user agents and is useful for log matching.

Does OnCrawl crawl for AI search or training?

No such purpose is documented. Its requests support customer-configured technical SEO audits.

OnCrawl & AI Search: What to Know

What is OnCrawl?

OnCrawl is the crawler in Oncrawl's enterprise technical SEO platform, which also includes log analysis. A customer creates a crawl profile for a site and chooses where the crawler starts. The resulting requests are part of that configured audit, not a public search engine gathering pages for its own results.

A crawl can be limited with URL inclusion or exclusion patterns, selected subdomains, and start URLs. For a verified domain, Oncrawl also lets the project owner create a virtual robots.txt file that applies only to the Oncrawl run. That override can give the crawler access to a subset of pages even when the site's published robots file would otherwise prevent the audit.

This directory matches www.oncrawl.com inside recorded mobile-style user agents, while Oncrawl's own robots examples use the product token OnCrawl. Robots compliance is not confirmed in the local metadata, and Cloudflare's bot directory marks the bot as not following robots.txt. The operator's documented virtual override also means a public robots rule is not always the final crawl policy for a verified project.

OnCrawl is a technical SEO audit tool, not a documented AI search or training collector. Reports from it may help a site owner repair crawlability problems, but an OnCrawl visit does not place a URL into an AI answer index or signal that a model used the page.

Indirectly relevant

Is OnCrawl relevant for AI search?

Indirectly. OnCrawl has no AI product of its own, but its output can end up in the systems that AI answers draw on.

OnCrawl is an SEO auditing crawler. It has no direct effect on search rankings or AI answers, but the reports it feeds are used to fix crawlability and content issues that do affect how search and AI systems see your site.

How to handle OnCrawl

For an intentional audit, set the narrowest useful scope in the Oncrawl crawl profile and allow the crawler through the WAF only where needed. Review any virtual robots.txt configured in the project, since it can differ from the file served by the website.

The operator's documented product token gives this site-wide rule:

User-agent: OnCrawl
Disallow: /

Treat it as a published preference, not an access-control boundary. Cloudflare reports that the crawler does not follow robots.txt, and a verified Oncrawl project can apply a virtual override. Use authentication or an edge block when a route must remain inaccessible regardless of crawl settings.

No verified IP status or request-signature directory is attached to this bot. Do not grant private access from the user agent alone.

Examples

An SEO team starts an OnCrawl audit at `/blog/` and uses URL inclusion rules so the crawler ignores the rest of the site.
A verified project uses a virtual robots.txt file to audit an approved staging section without changing the production robots policy.
A company with no Oncrawl project blocks the user agent at its CDN after checking that no agency initiated the crawl.

Frequently asked questions about OnCrawl

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Oncrawl operates the crawler as part of its technical SEO and log-analysis platform.

Related bots

AhrefsBot

Powers the database for both Ahrefs, a marketing intelligence platform, and Yep, an independent, privacy-focused search engine.

SEO

AhrefsSiteAudit

Powers Ahrefs’ Site Audit tool. Ahrefs users can use Site Audit to analyze websites and find both technical SEO and on-page SEO issues.

SEO

Barkrowler

Barkrowler is Babbar's web crawler that fuels and updates their graph representation of the web, providing SEO tools for the marketing community.

SEO

ClarityBot

ClarityBot is seoClarity's web crawler that performs technical SEO audits, analyzes content, and monitors website performance.

SEO

DataForSeoBot

DataForSeoBot is a backlink checker bot operated by DataForSEO that crawls websites to build and maintain their backlink database.

SEO

OnCrawl