Visible Amsterdam: Join us Sep 3
Promptwatch Logo

TNOThesisCrawler

Research crawler by TNO collecting publicly available academic theses from university websites for the DIAMONDS platform.
TNOTNOThesisCrawler
Academic Research

What is TNOThesisCrawler?

TNOThesisCrawler is TNO's research crawler. Research crawler by TNO collecting publicly available academic theses from university websites for the DIAMONDS platform.

TNOThesisCrawler collects pages for research rather than commercial indexing. Research corpora are a common starting point for public AI training datasets, so content it gathers can end up influencing what models know about your brand even though the crawl itself is academic.

Like any automated client, TNOThesisCrawler consumes crawl budget and appears in your server and CDN logs. Reviewing those logs alongside your robots.txt rules helps you keep automated traffic deliberate and easy to reason about.

Want to see every AI bot hitting your site? Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic, so you can watch ChatGPT, Claude, Perplexity, Gemini, and others crawl your pages and connect those visits to real citations and revenue. Learn more in AI crawler logs.

See every AI bot hitting your site

Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic. Watch ChatGPT, Claude, Perplexity, Gemini, and more crawl your pages in real time, see exactly what they take, and connect every crawl to the citations and revenue it drives.

How to handle TNOThesisCrawler

Allow TNOThesisCrawler if you are comfortable with your pages appearing in research datasets. Disallow it if you want to limit how far your content travels, including into datasets later used for AI training.

To control TNOThesisCrawler, add a rule for its user agent to your robots.txt:

User-agent: TNOThesisCrawler
Disallow: /

TNO does not publish a robots.txt commitment for TNOThesisCrawler, so confirm how it behaves by watching your access logs.

TNOThesisCrawler is a Cloudflare signed agent: it signs requests with Web Bot Auth and publishes its keys at https://diamonds.tno.nl/.well-known/http-message-signatures-directory, so you can tell genuine traffic from anything spoofing its user agent.

Examples

  • A publisher sees TNOThesisCrawler requesting full article text for a corpus study and decides whether to allow it.
  • A site owner disallows TNOThesisCrawler to keep subscriber-only content out of research datasets.

Related bots

Bibliothèque nationale de France Crawler

Bibliothèque nationale de France's mission is to collect, catalog, preserve, enrich and communicate the national documentary heritage.

Academic Research

CMU-Cylab

An academic research bot, conducting research on patterns in counterfeit document sales over the internet.

Academic Research

kb.dk_bot

Royal Danish Library collects the Danish Internet according to the Danish Legal Deposit Act for research purposes.

Academic Research

Library Of Congress Web Archiving

The Library of Congress Web Archive manages, preserves, and provides access to archived web content selected by subject experts from across the Library, so that it will be available for researchers today and in the future.

Academic Research

OMIM Litrack

OMIM Litrack is a literature tracking crawler.

Academic Research

Frequently asked questions about TNOThesisCrawler

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

TNOThesisCrawler is operated by TNO. It functions as TNO's research crawler.

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.

Promptwatch Dashboard