What is Diffbot?
Diffbot turns public web pages into structured records. Its crawler reads pages, identifies entities such as articles and organizations, and feeds a self-updating Knowledge Graph that customers can search or use to enrich their own data.
The Diffbot token denotes Diffbot's proactive crawl for its general search engine and Knowledge Graph. Diffbot documents a separate Diffbot-User agent for a URL fetched in response to a human using its software. Customer-run Extract and Crawl API jobs can also send a custom user agent, so not every request made through Diffbot infrastructure will carry the token covered on this page.
Diffbot says its proactive crawler uses caching, compression, conditional requests, and predictive scheduling to reduce repeat work on origin servers. Those measures describe normal behavior, not an identity check. this bot has no Cloudflare entry, verified IP range, or HTTP-signature directory for the crawler.
There is an important training distinction. Diffbot sells structured Knowledge Graph data for uses that include machine learning, retrieval, and data enrichment, but its crawler documentation says the Diffbot crawl is not used to train generative AI foundation models. Allowing it can make a page discoverable in Diffbot's graph and web-search services; it does not grant a foundation-model trainer direct access under this token.
By default, Diffbot follows Disallow and Crawl-delay in robots.txt. It does not follow the Allow directive. Diffbot also says robots rules may be overridden for a specific partnership or agreement, while customer Crawl API jobs can disable the default robots setting. The true metadata value therefore describes the standard proactive crawl, not every possible Diffbot-backed fetch.
A site should choose access based on whether it wants its public facts represented in Diffbot's products and downstream customer uses. If a page contains data that should not enter the Knowledge Graph, exclude it explicitly and watch for other Diffbot-related agents rather than assuming one rule covers customer-configured jobs.
