Does Diffbot-User respect robots.txt?

Diffbot says its crawls respect robots.txt by default, with exceptions where a customer has an agreement with the site.

Diffbot-User: AI Crawler Guide & robots.txt

What is Diffbot-User?

Diffbot-User is listed in Diffbot's robots.txt documentation next to Diffbot. The docs describe it as used by requests originating on behalf of a human user browsing a URL with Diffbot software, in response to their input.

Diffbot is the general, proactive crawler used to build a search engine and the Diffbot Knowledge Graph, and Diffbot says it is not used for AI training. Diffbot-User is a different identity for user-driven fetches.

Diffbot says its web crawls adhere to a site's robots.txt, including disallow and crawl-delay, though the rules can be overridden when a customer has an agreement with the site.

Relevant for AI search

Is Diffbot-User relevant for AI search?

Yes. Diffbot-User reads pages on behalf of people using Diffbot software, which feeds its AI and knowledge products.

Diffbot says Diffbot-User is used for requests that originate on behalf of a human user browsing a URL with Diffbot software. It is separate from Diffbot, the proactive crawler that builds Diffbot's search index and Knowledge Graph. Blocking Diffbot-User stops those user-driven reads and does not change whether the main crawler indexes you.

How to handle Diffbot-User

To block Diffbot-User:

User-agent: Diffbot-User
Disallow: /

To allow it while a catch-all rule blocks other bots, give it its own group with Allow: /. Diffbot does not publish IP ranges for this agent, so rely on logs and rate limits. Manage the Diffbot crawler with its own group.

Examples

A Diffbot customer opens a URL in Diffbot software to extract structured data, and the request arrives as Diffbot-User.
A site owner allows Diffbot-User but blocks `Diffbot` to stay out of the Knowledge Graph crawl.

Frequently asked questions about Diffbot-User

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

`Diffbot` crawls the web proactively. Diffbot-User fetches a URL when a person using Diffbot software asks for it.

Related bots

Diffbot

Diffbot crawls and structures web pages into a knowledge graph that is sold for AI training, retrieval, and data enrichment.

AI Crawler

FirecrawlAgent

Firecrawl's crawler, used by customers of its scraping API to convert websites into LLM-ready data.

AI Crawler

ChatGPT-User

Handles user-initiated requests in ChatGPT, accessing external content to provide real-time information; not used for automated crawling or AI training.

AI Assistant