Who operates Apify Website Content Crawler?

Apify Website Content Crawler is operated by Apify. It functions as Apify's AI agent.

Does Apify Website Content Crawler respect robots.txt?

Apify has not published a robots.txt commitment for Apify Website Content Crawler. Treat its behavior as unconfirmed and watch your access logs to see whether your rules are honored.

Should I block Apify Website Content Crawler?

Usually no. Apify Website Content Crawler visits because someone asked it to complete a task, so blocking it can break that for real users. Block it only if you do not want agents transacting on your site.

How can I verify Apify Website Content Crawler is genuine?

Apify Website Content Crawler is a Cloudflare signed agent. It signs requests using Web Bot Auth HTTP message signatures and publishes its public keys at https://api.apify.com/.well-known/http-message-signatures-directory, so you can validate a request at your edge instead of trusting the user-agent string, which anyone can spoof.

Apify Website Content Crawler - What It Is & How to Handle It

What is Apify Website Content Crawler?

Apify Website Content Crawler is Apify's AI agent. Crawl websites and extract content to feed AI apps. Convert web data to Markdown or HTML, download files, and more.

Apify Website Content Crawler acts on behalf of a person rather than indexing the web in bulk. It arrives when someone asks it to complete a task on your site, so the pages and flows it can reach decide whether that task succeeds. Blocking it will not remove you from AI answers, but it can stop real people from finishing real actions.

Tracking which AI crawlers and agents reach your site, and what they do once there, is the foundation of generative engine optimization. See our guides to AI crawlers and robots.txt to control automated access and protect your AI search visibility.

Want to see every AI bot hitting your site? Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic, so you can watch ChatGPT, Claude, Perplexity, Gemini, and others crawl your pages and connect those visits to real citations and revenue. Learn more in AI crawler logs.

See every AI bot hitting your site

Promptwatch turns your server and CDN logs into a live view of AI crawler and agent traffic. Watch ChatGPT, Claude, Perplexity, Gemini, and more crawl your pages in real time, see exactly what they take, and connect every crawl to the citations and revenue it drives.

How to handle Apify Website Content Crawler

Most sites keep Apify Website Content Crawler allowed, because it shows up when a real person asks it to do something on their behalf. Block it only if you do not want automated agents acting on your site.

To control Apify Website Content Crawler, add a rule for its user agent to your robots.txt:

User-agent: ApifyWebsiteContentCrawler
Disallow: /

Apify does not publish a robots.txt commitment for Apify Website Content Crawler, so confirm how it behaves by watching your access logs.

Apify Website Content Crawler is a Cloudflare signed agent: it signs requests with Web Bot Auth and publishes its keys at https://api.apify.com/.well-known/http-message-signatures-directory, so you can tell genuine traffic from anything spoofing its user agent.

Examples

A customer sends Apify Website Content Crawler to complete a task on a site, and the operations team confirms the agent reached the pages it needed.
An engineer verifies Apify Website Content Crawler against its published signature keys before allowing it through the edge.

Related bots

Amazon Kendra

Amazon Kendra is a managed information retrieval and intelligent search service that uses natural language processing and advanced deep learning model.

AI Assistant