How broad is Determ's source coverage?

Determ says Mediatoolkitbot searches more than 100 million online sources. That is a service-wide figure, not a promise that every page on a particular site will be visited.

Does Mediatoolkitbot follow robots.txt?

Yes. Determ documents both `Disallow` and `Crawl-delay` for the exact token. A malformed robots file may not produce the result the site owner expects.

What changes when a site blocks the crawler?

Determ may stop finding mentions on the blocked pages. Customers tracking phrases that appear there can then receive incomplete coverage.

Is Mediatoolkitbot an AI search or training crawler?

No such use is documented. Its stated purpose is phrase matching for media monitoring, not generative answers or model training.

Mediatoolkitbot & AI Search: What to Know

What is Mediatoolkitbot?

A mention in a Determ report can start with a visit from Mediatoolkitbot. The crawler reads public web content and compares it with phrases that Determ customers have chosen to monitor. The customer defines each phrase used for matching.

Determ says the service searches more than 100 million online sources. When text matches a tracked phrase, Determ can show that page as a mention to the relevant customer for reporting or follow-up. The crawler is therefore looking for monitored references, not building a general consumer search index.

The recorded header is Mediatoolkitbot ([email protected]). Determ explicitly documents robots.txt support and accepts Crawl-delay for this token. Its crawler page also warns that an invalid robots file may keep the bot from recognizing the intended rule.

This crawl can affect whether a public mention is found in Determ. It has no documented role in generative search and is not described as a source of language-model training data. Blocking it can create a gap in media-monitoring results, but it does not control a site's presence in AI answers.

Indirectly relevant

Is Mediatoolkitbot relevant for AI search?

Indirectly. Mediatoolkitbot has no AI product of its own, but its output can end up in the systems that AI answers draw on.

Mediatoolkitbot collects content from many sites and redistributes it through its own product. Aggregated copies can end up in datasets that AI systems later learn from, and some aggregators are themselves sources that AI answer engines draw on.

How to handle Mediatoolkitbot

Publishers and organizations that want public references detected by Determ can leave article pages open. If the crawler is simply too frequent, set a delay for Mediatoolkitbot and measure the result before removing the domain from monitoring.

Determ documents this opt-out:

User-agent: Mediatoolkitbot
Disallow: /

Validate the finished robots file because Determ says syntax errors can make a directive unreadable to the bot. A complete block prevents later mentions on those pages from reaching Determ customers. For behavior that does not match the documented rule, use the contact address on Determ's current crawler page.

Examples

An industry journal wants brands to find its coverage through media monitoring, so it leaves published articles open and adds a crawl delay for its slower archive.
A members-only research area should never be part of public mention tracking, so the site keeps authentication in place and disallows `Mediatoolkitbot` on that path as an extra signal.

Frequently asked questions about Mediatoolkitbot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Determ crawls public sources and checks their text against phrases selected by its customers. A match can appear as a mention in that customer's monitoring results.