Does SiteGuru respect robots.txt?

Yes. SiteGuru generally follows robots.txt directives, so you can allow or disallow it with a rule for its user agent.

Should I block SiteGuru?

It depends on whether you rely on the service behind SiteGuru. If your team or your agency uses it, keep it allowed; otherwise you can disallow it in robots.txt to save crawl budget.

Visible Amsterdam: Join us Sep 3Visible Amsterdam: Join us on September 3 for an exclusive afternoon exploring the future of AI Search. Request your spot now

Book a Demo

SiteGuru

SiteGuru is an SEO auditing tool that crawls.

SiteGuruSiteGuruCrawler

SEO

What is SiteGuru?

SiteGuru is a SEO crawler. SiteGuru is an SEO auditing tool that crawls.

SiteGuru is an operational bot rather than an AI crawler, so it does not directly affect whether you appear in AI answers. It still shows up in your logs, and bots like it help keep your site healthy and crawlable, which indirectly supports both SEO and AI visibility.

Like any automated client, SiteGuru consumes crawl budget and appears in your server and CDN logs. Reviewing those logs alongside your robots.txt rules helps you keep automated traffic deliberate and easy to reason about.

Want to see every AI bot hitting your site? Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic, so you can watch ChatGPT, Claude, Perplexity, Gemini, and others crawl your pages and connect those visits to real citations and revenue. Learn more in AI crawler logs.

See every AI bot hitting your site

Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic. Watch ChatGPT, Claude, Perplexity, Gemini, and more crawl your pages in real time, see exactly what they take, and connect every crawl to the citations and revenue it drives.

How to handle SiteGuru

Whether to allow SiteGuru depends on whether you rely on the service behind it. Use a robots.txt rule to allow or disallow it.

To control SiteGuru, add a rule for its user agent to your robots.txt:

User-agent: SiteGuruCrawler
Disallow: /

SiteGuru generally honors robots.txt directives.

Examples

A site owner spots SiteGuru in access logs and confirms it against the operator's documented user agent.
An operations team allows SiteGuru so the SEO reporting it powers stays accurate.

Related bots

AhrefsBot

Powers the database for both Ahrefs, a marketing intelligence platform, and Yep, an independent, privacy-focused search engine.

SEO

AhrefsSiteAudit

Powers Ahrefs’ Site Audit tool. Ahrefs users can use Site Audit to analyze websites and find both technical SEO and on-page SEO issues.

SEO

Barkrowler

Barkrowler is Babbar's web crawler that fuels and updates their graph representation of the web, providing SEO tools for the marketing community.

SEO

ClarityBot

ClarityBot is seoClarity's web crawler that performs technical SEO audits, analyzes content, and monitors website performance.

SEO

DataForSeoBot

DataForSeoBot is a backlink checker bot operated by DataForSEO that crawls websites to build and maintain their backlink database.

SEO

Frequently asked questions about SiteGuru

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

SiteGuru is operated by the company of the same name. It functions as a SEO crawler.

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.