Does YouBot respect robots.txt?

You.com's documentation says yes, including user-agent rules and crawl-delay, and this page records the value as true. Cloudflare's bot entry says false, so site owners with strict requirements should test the rule against signed traffic.

How quickly does a YouBot robots.txt change apply?

You.com says YouBot caches robots.txt for 30 minutes. A change can therefore take up to 30 minutes to affect its crawling.

Does blocking YouBot opt my content out of AI training?

No such training role is documented for this crawler. The rule stops YouBot's search discovery on compliant requests; it should not be described as a broader training opt-out.

Does allowing YouBot guarantee a You.com citation?

No. It lets You.com crawl and index the page, but the search system still decides which sources answer a particular query.

YouBot: AI Crawler Guide & robots.txt

Q: How can I verify YouBot?

The recommended method is Web Bot Auth signature validation using https://you.com/.well-known/http-message-signatures-directory. You.com also documents reverse DNS and the 68.67.112.0/24 source range.

What is YouBot?

YouBot is the crawler behind the You.com search engine. It discovers and indexes public pages so You.com can return current search results and cite web sources in AI answers. Its documented identity is YouBot/1.0 within a Chrome-style user-agent string.

The crawler can index HTML, PDFs, plain text, XML and RSS feeds, plus structured data embedded in pages. This is recurring search-engine crawling rather than a browser agent completing a transaction for one user. A visit usually means You.com is discovering or refreshing material for its search system.

You.com's YouBot documentation says the crawler adapts its rate, honors crawl-delay, and responds to server status codes. A 429 response reduces its request rate, while a 503 pauses crawling. The documentation says robots.txt is cached for 30 minutes, so a rule change may not appear immediately.

Indexing can make a page available for a You.com result or citation, though access alone does not guarantee placement. Disallowing YouBot prevents future compliant crawls of the blocked paths and can leave You.com without the latest version of that content.

No model-training use is documented for YouBot. Its stated job is search discovery and indexing. A rule aimed at this token controls the You.com search crawler, not a general opt-out from AI training by You.com or any other company.

You.com recommends Web Bot Auth for identity checks. Signed requests can be validated with the Ed25519 public keys at https://you.com/.well-known/http-message-signatures-directory. Its documentation also gives the reverse-DNS form youbot-{address}.search.you.com and publishes 68.67.112.0/24 as the source range, although the bot's listing does not mark YouBot as IP verified.

There is one source conflict to account for. You.com's documentation and this bot's metadata say YouBot respects robots.txt, but Cloudflare's bot record marks followsRobotsTxt false. Use the published robots rule as the operator-supported control, then check logs if compliance is important. The Web Bot Auth signature remains the stronger identity check.

Relevant for AI searchYou.com

Is YouBot relevant for AI search?

Yes. YouBot feeds You.com, so the pages it can reach shape what those AI products say about you.

YouBot crawls and indexes pages so the AI search or assistant behind it can retrieve them at answer time. A page it has never fetched cannot be quoted, summarized, or linked in that product's answers, so most sites keep it allowed to stay citable. Blocking it removes your pages from that AI surface and hands those citations to competitors.

How to handle YouBot

Keep YouBot allowed on pages you want You.com to discover and refresh. Use a targeted or site-wide disallow when you do not want those URLs indexed by this crawler.

You.com publishes this robots.txt control:

User-agent: YouBot
Disallow: /

Its official documentation says the crawler follows robots directives and may cache the file for 30 minutes. Because Cloudflare's registry carries a conflicting false flag, verify the result in access logs rather than assuming the two records agree.

Authenticate a claimed YouBot request by validating its Web Bot Auth signature against https://you.com/.well-known/http-message-signatures-directory. Reverse DNS and the published 68.67.112.0/24 range are secondary checks. Do not trust YouBot/1.0 by itself.

Examples

A publisher returns 429 while its origin is under load, and YouBot reduces its crawl rate according to You.com's documented behavior.
A product team leaves public documentation open to YouBot so revised pages can replace older copies in You.com's index.
A network engineer validates an Ed25519 Web Bot Auth signature before treating a Chrome-like request as genuine YouBot traffic.

Frequently asked questions about YouBot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

You.com says it handles HTML pages, PDFs, text files, XML and RSS feeds, and structured data. The indexed material supports You.com search results and cited AI answers.