Does Sitebulb respect robots.txt?

It does by default, but a user can disable that setting or replace the live file with a virtual one for the audit. The record therefore marks support as partial.

Why might Sitebulb traffic look like Googlebot or a browser?

Audit settings let the user choose preset search-engine and browser identities or enter a custom user agent. The request header is part of the test configuration.

Does a Sitebulb crawl contribute to AI answers?

The crawl supplies a private technical SEO audit. Sitebulb does not present it as public answer retrieval or a model-training feed.

Can I authenticate Sitebulb from the `Sitebulb` token?

No. This entry is marked unverifiable, and the software permits identity changes. Confirm an expected audit with its operator or use a unique custom string and ordinary access controls.

Sitebulb & AI Search: What to Know

What is Sitebulb?

Sitebulb crawls a website only after a user chooses that site for an audit in Sitebulb Desktop or Sitebulb Cloud. It follows links to inspect pages and feeds the results into a private technical SEO report. This is a user-started audit, but it is not an AI assistant retrieving a page for an answer.

Desktop and mobile audits have different default request strings. The desktop form contains Sitebulb/1.1, while the smartphone form is browser-like and points to sitebulb.com. A user can instead select a search-engine identity or set another user agent, so the Sitebulb token will not identify every crawl made with the software.

Sitebulb respects robots directives in its default configuration. The audit owner can untick that setting or supply a virtual robots.txt file that replaces the live one for the crawl. This configurable override is why the record describes compliance as partial rather than treating robots.txt as guaranteed enforcement.

Crawled content is used to diagnose the selected site's SEO condition. Sitebulb's documentation does not assign this bot a public search, AI citation, or model-training function. Allowing an authorized audit can help its user find crawlability problems, but a Sitebulb visit has no direct effect on whether an answer engine cites the page.

Indirectly relevant

Is Sitebulb relevant for AI search?

Indirectly. Sitebulb has no AI product of its own, but its output can end up in the systems that AI answers draw on.

Sitebulb is an SEO auditing crawler. It has no direct effect on search rankings or AI answers, but the reports it feeds are used to fix crawlability and content issues that do affect how search and AI systems see your site.

How to handle Sitebulb

Coordinate an expected Sitebulb audit with the person running it. Ask which user agent, crawl speed, and robots settings they selected, especially for a staging site or a crawl that needs JavaScript rendering.

A policy for the recorded product token is:

User-agent: Sitebulb
Disallow: /

Sitebulb follows the rule by default, but its robots support is partial because the user can override the live file. Check the logs for the actual request identity. If the content must not be fetched, require authentication or enforce a block at the server or edge rather than relying on robots.txt.

Do not broadly allow every request containing a browser or search-engine identity just to make an audit work. Sitebulb supports a custom user agent, which lets the audit owner choose a distinct string that can be narrowly allowed and recognized.

Examples

An agency schedules a Sitebulb Cloud audit and gives the client the exact custom user agent that will appear in the CDN logs.
A developer uses a virtual robots.txt inside Sitebulb to test a proposed rule without changing the live file.
A publisher sees the default desktop token but has authorized no audit, so it blocks the traffic at the edge.

Frequently asked questions about Sitebulb

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

No. Sitebulb says its bot crawls only when a user proactively selects a website for an audit.

Related bots

Lumar

Lumar runs cloud-based crawls for sites selected by its customers and turns the fetched page data into private website analytics.

SEO

ScreamingFrogBot

Screaming Frog SEO Spider is a website crawler used by SEO professionals for site audits and technical SEO analysis.

SEO

AhrefsBot

Powers the database for both Ahrefs, a marketing intelligence platform, and Yep, an independent, privacy-focused search engine.

SEO

AhrefsSiteAudit

Powers Ahrefs’ Site Audit tool. Ahrefs users can use Site Audit to analyze websites and find both technical SEO and on-page SEO issues.

SEO

Barkrowler

Barkrowler is Babbar's web crawler that fuels and updates their graph representation of the web, providing SEO tools for the marketing community.

SEO

Sitebulb