What does a full SiteGuru audit inspect?

SiteGuru names HTML checks for titles, meta descriptions, and missing alt text, along with W3 validation and PageSpeed analysis for the monitored site.

How does SiteGuru discover pages?

The operator says it crawls URLs found in the sitemap or linked from another page. It does not submit forms.

Does SiteGuru respect robots.txt?

Yes. SiteGuru provides explicit allow and disallow examples for `SiteGuruCrawler`, and this bot marks support as confirmed.

Will SiteGuru overload a server?

The operator says it limits concurrent requests. If traffic still causes pressure, inspect the request rate and apply a suitable edge limit.

Does SiteGuru feed AI search or model training?

SiteGuru describes the crawl as private SEO monitoring and broken-link checking, with no AI search or model-training destination.

SiteGuru & AI Search: What to Know

What is SiteGuru?

SiteGuruCrawler visits for two documented reasons. Someone may have added the site to SiteGuru for ongoing SEO monitoring, or a monitored site may link to the URL and SiteGuru is checking whether that external link works. The second case explains visits to sites that have no SiteGuru account.

For a full site audit, SiteGuru reads public HTML to inspect page titles, meta descriptions, and missing image alt text. It also uses the W3 Validator and Google's PageSpeed API when checking the monitored site. An external broken-link check is narrower and does not run those full-site tests.

The crawler discovers pages from the site's sitemap and from links on crawled pages. SiteGuru says it does not submit forms and limits concurrent requests to reduce load. Its data goes to the site owner's SEO report rather than a public index.

SiteGuru states that SiteGuruCrawler respects robots.txt, matching the confirmed value in this bot. The service does not document an AI answer-retrieval or model-training use for these fetches. An audit may help its user fix pages that other crawlers struggle with, but SiteGuru itself does not determine AI citations.

Indirectly relevant

Is SiteGuru relevant for AI search?

Indirectly. SiteGuru has no AI product of its own, but its output can end up in the systems that AI answers draw on.

SiteGuru is an SEO auditing crawler. It has no direct effect on search rankings or AI answers, but the reports it feeds are used to fix crawlability and content issues that do affect how search and AI systems see your site.

How to handle SiteGuru

Keep SiteGuruCrawler available if the site owner uses SiteGuru monitoring or wants linked pages checked for broken links. If neither use is wanted, the operator documents this block:

User-agent: SiteGuruCrawler
Disallow: /

SiteGuru says it will respect the rule, and this bot confirms robots.txt support. A blank Disallow: can be used under the same group when the crawler should be allowed throughout the site.

Do not expose private pages on the assumption that the crawler will avoid them. Keep protected content behind authentication. The record provides no verified IP range or request signature, so use the token to classify traffic rather than authenticate it.

Examples

A store owner adds the domain to SiteGuru and sees `SiteGuruCrawler` follow sitemap and internal links during the initial audit.
A small blog receives one SiteGuru request because a monitored customer site links to one of its posts and the service checks for a broken link.
A private documentation site blocks SiteGuruCrawler in robots.txt and retains login protection for every restricted page.

Frequently asked questions about SiteGuru

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

A different site being monitored by SiteGuru may link to it. SiteGuru follows that external link to check whether it is broken.

Related bots

AhrefsBot

Powers the database for both Ahrefs, a marketing intelligence platform, and Yep, an independent, privacy-focused search engine.

SEO

AhrefsSiteAudit

Powers Ahrefs’ Site Audit tool. Ahrefs users can use Site Audit to analyze websites and find both technical SEO and on-page SEO issues.

SEO

Barkrowler

Barkrowler is Babbar's web crawler that fuels and updates their graph representation of the web, providing SEO tools for the marketing community.

SEO

ClarityBot

ClarityBot is seoClarity's web crawler that performs technical SEO audits, analyzes content, and monitors website performance.

SEO

DataForSeoBot

DataForSeoBot is a backlink checker bot operated by DataForSEO that crawls websites to build and maintain their backlink database.

SEO

SiteGuru