Does Applebot respect robots.txt?

Yes for general search crawling. Apple says it follows Applebot-specific rules, falls back to Googlebot rules when no Applebot group is present, and does not support crawl-delay.

What is Applebot-Extended?

It is a robots.txt policy token for controlling whether Apple may use Applebot-crawled content to train general-purpose foundation models. It does not make its own page requests, and disallowing it does not remove the site from Apple search.

How do I keep a page out of Apple's generated-answer context?

Apple documents the `nosnippet` robots meta directive for that purpose. The page can still appear as a title-only suggestion, but Apple says the tagged content will not be used as current context for generated output.

Can Applebot crawl paywalled content?

Apple says Applebot does not crawl content that requires login credentials or is protected by a paywall. It also recognizes `isAccessibleForFree: false` and excludes that content from extra generated-answer context while retaining search eligibility.

How can I verify Applebot traffic?

Use forward-confirmed reverse DNS for a hostname under applebot.apple.com or compare the source with Apple's published IP CIDR data. The user-agent token alone is not proof.

Applebot: AI Crawler Guide & robots.txt

What is Applebot?

Applebot is Apple's crawler for search technology used across Spotlight, Siri, Safari, and other Apple experiences. Allowing it makes a site eligible to appear in those search surfaces, although Apple still ranks and selects results independently.

The crawler can render a page in a browser, so blocked JavaScript, CSS, XHR, or other required resources may leave it with an incomplete version. Apple says the crawl rate adapts when a server slows down or returns errors, and cached content helps reduce repeat requests.

Apple may also use crawled pages as current context when an AI model generates an answer in an Apple product. Those broad world-knowledge answers can include source links. A page-level nosnippet directive keeps the tagged content out of that generated context while leaving a title-only search suggestion possible.

Model training is controlled separately. Apple says data gathered by Applebot may help train foundation models for Apple Intelligence, Services, and Developer Tools. Publishers can disallow Applebot-Extended to opt out of that training use. Applebot-Extended does not fetch pages; it tells Apple how data already crawled by Applebot may be used.

Apple also recognizes the schema.org isAccessibleForFree: false property for paywalled or subscription pages. Such a page can remain eligible for search, but Apple says it will not use the paywalled content as extra context for generated output. Applebot itself does not crawl sites that require login credentials or are protected by a paywall.

For general crawling, Applebot follows standard robots.txt allow and disallow rules. If no Applebot group exists but a Googlebot group does, Apple says it follows the Googlebot instructions. It does not honor crawl-delay, so server responses and rate controls are still useful for load management.

The stable token is Applebot, and the full user agent includes the Applebot information URL recorded by Cloudflare. Apple documents reverse DNS under *.applebot.apple.com and publishes IP CIDR data for verification. Cloudflare's bot directory marks the bot verified and robots compliant, but the bot's listing does not mark it as IP verified.

Relevant for AI searchSiriApple IntelligenceSpotlight

Is Applebot relevant for AI search?

Yes. Applebot feeds Siri, Apple Intelligence, Spotlight, so the pages it can reach shape what those AI products say about you.

Applebot gathers public web content that can end up in the training data for large language models. Once your pages are in that set, they influence how the operator's models talk about you for that model generation. Allowing it lets your own writing carry weight in those answers; blocking it means the models learn about you from third parties instead.

How to handle Applebot

Choose the control that matches the intended outcome. To stop Applebot from crawling the entire site for search, current AI context, and possible later use, publish:

User-agent: Applebot
Disallow: /

To keep Apple search discovery while opting the crawled content out of general-purpose foundation-model training, use the separate policy token:

User-agent: Applebot-Extended
Disallow: /

Applebot-Extended does not crawl. It only controls training use of data fetched by Applebot. For a page that may stay in search but should not supply text to broad world-knowledge answers, Apple documents nosnippet as the relevant page-level directive.

Applebot generally follows robots.txt, but it does not follow crawl-delay. Verify a claimed crawler through forward-confirmed reverse DNS in applebot.apple.com or Apple's published CIDR file before granting any special network treatment.

Examples

A publisher allows Applebot for Spotlight and Safari discovery but disallows Applebot-Extended across the site to decline foundation-model training use.
A reference site applies `nosnippet` to one licensed article so Apple can retain a title-only result without using the article as generated-answer context.
A developer unblocks the JavaScript and CSS required to render a public product page after finding that Applebot saw an incomplete document.

Frequently asked questions about Applebot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Apple operates Applebot. the operator is not listed in the bot directory, while Apple's support documentation and Cloudflare's verified-bot record identify Apple as the operator.