What is Novellum AI Crawl?
Novellum AI Crawl is a site-fetching tool for agents built by Novellum. Cloudflare's directory points to Novellum's crawler documentation and describes the service as an MCP tool that agents use to crawl sites.
MCP is the interface through which an agent can call the crawler during a task. The supplied record does not say whether every request comes directly from a person, from an autonomous workflow, or from a scheduled job. A log entry proves only that Novellum's crawling infrastructure claims to have fetched the URL.
No available description says that Novellum AI Crawl builds a public search index or gathers a model-training corpus. Its documented role is retrieval for agents. A visit can provide page content to the agent using the tool, but it is not evidence of search ranking, citation eligibility, or training use.
The recorded user agent and matching token are both Novellum. That short string is easy to filter, but it is also easy for an unrelated client to copy. User-agent matching is useful for policy and reporting, not for authentication.
Robots.txt behavior is unknown in the page metadata. Cloudflare explicitly marks Novellum AI Crawl as not following robots.txt, so a Disallow group should be treated as a published preference rather than an access control.
Novellum supplies a Web Bot Auth key directory at https://crawl.corp.novellum.ai/.well-known/http-message-signatures-directory. A valid HTTP message signature lets an edge verify the request against Novellum's published key instead of trusting the user agent. That proves infrastructure identity; the site still decides whether the agent may read a particular route.
