What does the crawler save from a page?

It can clean the page and save plain text, Markdown, or HTML. Depending on the run settings, it can also keep screenshots, metadata, and linked document files in the run's Apify storage.

Does Apify Website Content Crawler respect robots.txt?

Its robots.txt behavior is unknown in this catalog, and Cloudflare's directory marks the agent as not following robots.txt. Keep the bot-specific rule as a stated preference, but use an edge rule when the outcome must be enforced.

Does a visit mean my content is being used for AI training?

No conclusion about training follows from the visit. The Actor prepares web content for many possible uses, including retrieval and LLM applications, and the customer running it controls the downstream use.

How can I authenticate this Apify agent?

Verify its Web Bot Auth HTTP message signature with the keys at `https://api.apify.com/.well-known/http-message-signatures-directory`. Matching `ApifyWebsiteContentCrawler` without a valid signature is not sufficient proof.

Apify Website Content Crawler AI Crawler Guide

What is Apify Website Content Crawler?

Apify Website Content Crawler is an Apify Actor for turning a website into reusable text and files. A customer supplies one or more start URLs, and the Actor follows links below that URL path. It is commonly run against documentation, a knowledge base, a blog, or another text-heavy section rather than sent out to discover the public web on its own.

After loading a page, the Actor can remove navigation, headers, cookie notices, and other elements before saving the main content as plain text, Markdown, or HTML. It can also download linked documents when that option is enabled. Results are stored in an Apify dataset for export or use through an API.

Genuine signed traffic presents the token ApifyWebsiteContentCrawler, with Cloudflare recording the full user agent as ApifyWebsiteContentCrawler/1.0 (+https://apify.com/apify/website-content-crawler). Apify registers this agent for Web Bot Auth and publishes verification keys at https://api.apify.com/.well-known/http-message-signatures-directory. Signature validation is stronger evidence than the header alone.

The crawler can use raw HTTP for simpler pages or a headless browser for sites that need JavaScript rendering. Its scope is set by the person running it through start URLs, depth and page limits, and include or exclude patterns. That setup means two visits from this user agent may represent very different jobs, from one small documentation section to a large recursive crawl.

Apify has not published a stable robots.txt commitment for this Actor in the documentation reviewed here, so its behavior is classified as unknown. Cloudflare's directory lists it as not following robots.txt. A robots rule can still state a site owner's preference, but sites that require enforcement should use the verified signature with an edge access policy.

The extracted material is often fed to vector databases, retrieval systems, custom GPTs, or other LLM applications. That can affect answers produced by the customer's application, but it does not add a page to a public AI search index by itself. A request also does not prove that the content will train a model; the Apify customer chooses what happens to the resulting dataset.

Relevant for AI search

Is Apify Website Content Crawler relevant for AI search?

Yes. Apify Website Content Crawler collects pages for an AI product, so what it can crawl influences how AI systems describe your brand.

Apify Website Content Crawler crawls and indexes pages so the AI search or assistant behind it can retrieve them at answer time. A page it has never fetched cannot be quoted, summarized, or linked in that product's answers, so most sites keep it allowed to stay citable. Blocking it removes your pages from that AI surface and hands those citations to competitors.

How to handle Apify Website Content Crawler

Allow the Actor when you expect customers or partners to turn your public material into a dataset. Restrict private paths, expensive endpoints, and files that should not be collected regardless of who started the run. A valid Apify signature proves the request's identity, but it does not grant permission to every URL.

Publish this rule if you do not want the Actor to crawl the site:

User-agent: ApifyWebsiteContentCrawler
Disallow: /

Robots.txt behavior remains unconfirmed, and Cloudflare does not list this agent as following the file. For a dependable allow or deny decision, validate its Web Bot Auth signature against https://api.apify.com/.well-known/http-message-signatures-directory and apply the result at the CDN, firewall, or application boundary.

Examples

A support team crawls its public help center into Markdown, then loads that dataset into a retrieval system for its customer assistant.
A site owner allows product documentation but blocks a large download directory at the edge before an Apify run reaches it.
A platform engineer validates the Web Bot Auth signature on `ApifyWebsiteContentCrawler/1.0` requests instead of allowlisting a spoofable string.

Frequently asked questions about Apify Website Content Crawler

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

An Apify customer or an application using Apify starts the Actor with one or more URLs and crawl settings. The agent is not a general search crawler that chooses sites independently.

Related bots

Amazon Kendra

Amazon Kendra is a managed information retrieval and intelligent search service that uses natural language processing and advanced deep learning model.

AI Assistant