Visible Amsterdam: Join us Sep 3
Promptwatch Logo

Novellum AI Crawl

Novellum.ai is building out tools for building agents. This MCP tool will be used by agents to crawl sites.
NovellumNovellum
AI Crawler

What is Novellum AI Crawl?

Novellum AI Crawl is Novellum's AI crawler. Novellum.ai is building out tools for building agents. This MCP tool will be used by agents to crawl sites.

Novellum AI Crawl matters for AI visibility because the pages it collects can shape what AI systems learn about your brand, products, and expertise. Allowing it can strengthen how accurately those systems describe and recommend you, while disallowing it keeps your content out of their reach. Either way, knowing Novellum AI Crawl visits is the first step to managing how your brand shows up in AI search.

Tracking which AI crawlers and agents reach your site, and what they do once there, is the foundation of generative engine optimization. See our guides to AI crawlers and robots.txt to control automated access and protect your AI search visibility.

Want to see every AI bot hitting your site? Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic, so you can watch ChatGPT, Claude, Perplexity, Gemini, and others crawl your pages and connect those visits to real citations and revenue. Learn more in AI crawler logs.

See every AI bot hitting your site

Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic. Watch ChatGPT, Claude, Perplexity, Gemini, and more crawl your pages in real time, see exactly what they take, and connect every crawl to the citations and revenue it drives.

How to handle Novellum AI Crawl

Whether to allow Novellum AI Crawl depends on whether you or your customers rely on the service behind it. Use a robots.txt rule to allow or disallow it.

To control Novellum AI Crawl, add a rule for its user agent to your robots.txt:

User-agent: Novellum
Disallow: /

Novellum does not publish a robots.txt commitment for Novellum AI Crawl, so confirm how it behaves by watching your access logs.

Novellum AI Crawl is a Cloudflare signed agent: it signs requests with Web Bot Auth and publishes its keys at https://crawl.corp.novellum.ai/.well-known/http-message-signatures-directory, so you can tell genuine traffic from anything spoofing its user agent.

Examples

  • A publisher reviews server logs, sees Novellum AI Crawl requesting long-form articles, and decides whether to allow or disallow it in robots.txt.
  • A site owner adds a Disallow rule for Novellum AI Crawl to keep premium guides out of its dataset.

Related bots

AI2Bot

AI2Bot is operated by the Allen Institute for Artificial Intelligence (Ai2) to crawl the web for content to train open-source AI models.

AI Crawler

aiHitBot

aiHitBot collects and maintains historical information about companies.

AI Crawler

Amazonbot

Amazonbot is Amazon's web crawler used to improve our services, such as enabling Alexa to more accurately answer questions for customers.

AI Crawler

Applebot

Applebot powers search features in Apple's ecosystem (Spotlight, Siri, Safari) and may be used to train Apple's foundation models for generative AI features.

AI Crawler

Frequently asked questions about Novellum AI Crawl

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Novellum AI Crawl is operated by Novellum. It functions as Novellum's AI crawler.

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.

Promptwatch Dashboard