Does Amazon Bedrock Bot respect robots.txt?

Yes. Amazon says it follows RFC 9309, checks a crawler-specific bedrockbot rule before the generic token, and defaults to disallow if robots.txt is missing.

Does bedrockbot train Amazon foundation models?

Amazon documents the crawler as a knowledge-base ingestion tool. The supplied sources do not say that fetched pages are used to train foundation models.

Why is bedrockbot crawling my site?

An Amazon Bedrock customer configured one of your URLs as a knowledge-base data source. Amazon's terms require that customer to own the pages or have authorization to crawl them.

Can I allow one Bedrock crawler but block others?

Yes. Use the full crawler-specific bedrockbot identifier in robots.txt for the authorized job and keep a separate generic `bedrockbot` rule for all other jobs.

Amazon Bedrock Bot AI Crawler Guide

What is Amazon Bedrock Bot?

Amazon Bedrock Bot runs when an Amazon Bedrock customer configures a website as a data source for a knowledge base. The customer supplies one or more seed URLs and confirms that it is authorized to crawl them. This is a configured ingestion job, not Amazon continuously indexing the public web on its own behalf.

Starting from those URLs, the crawler follows links within the scope selected by the customer. Amazon supports path, host, or subdomain scope along with inclusion and exclusion filters. Customers can also cap the crawl rate and maximum page count, so request volume depends on the data-source configuration.

Retrieved pages are ingested into that customer's Bedrock knowledge base. A Bedrock application can then retrieve the material when answering questions. Access to the ingested content follows the customer's AWS permissions, which is different from placing the page in a public search engine.

Amazon documents this connector for retrieval, not foundation-model training. A bedrockbot request does not mean the page is being added to an Amazon model's training corpus. Its practical effect is limited to knowledge bases whose owners selected the site and can use the resulting data.

The user agent contains bedrockbot, a crawler-specific UUID, and an optional suffix. When reading robots.txt, Amazon checks the specific crawler rule before falling back to the generic bedrockbot token. This lets a site distinguish one authorized Bedrock ingestion job from other customers' jobs when the complete identifier is known.

Amazon says the crawler follows RFC 9309 robots.txt rules and defaults to disallow when it cannot find a robots.txt file. The supplied facts do not include Cloudflare operator data, published IP verification, or a Web Bot Auth directory. Use the UUID and expected crawl timing as supporting evidence, but do not treat the user agent as authentication.

Relevant for AI search

Is Amazon Bedrock Bot relevant for AI search?

Yes. Amazon Bedrock Bot collects pages for an AI product, so what it can crawl influences how AI systems describe your brand.

Amazon Bedrock Bot crawls and indexes pages so the AI search or assistant behind it can retrieve them at answer time. A page it has never fetched cannot be quoted, summarized, or linked in that product's answers, so most sites keep it allowed to stay citable. Blocking it removes your pages from that AI surface and hands those citations to competitors.

How to handle Amazon Bedrock Bot

Allow the crawler only for content that an authorized Bedrock knowledge-base owner should ingest. If your organization runs the knowledge base, prefer a narrow source path and crawler-specific rule. Other sites can use the generic token to refuse all Bedrock web-crawler jobs.

A generic site-wide block is:

User-agent: bedrockbot
Disallow: /

Amazon says the crawler obeys this rule. A more specific rule for the full bedrockbot UUID takes precedence when you need to allow one customer configuration. Keep private pages behind authentication because robots.txt controls crawling, not access authorization.

Examples

A software company authorizes a Bedrock knowledge base to ingest its public support manual and limits the seed URL to the documentation path.
A Bedrock customer sees an ingestion job fail because the source site has no robots.txt file, which Amazon treats as disallowed by default.
A site owner permits one known bedrockbot UUID while the generic rule blocks ingestion jobs configured by other AWS customers.

Frequently asked questions about Amazon Bedrock Bot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Amazon provides the crawler as part of the web data-source connector for Amazon Bedrock Knowledge Bases.

Related bots

Amazon Kendra

Amazon Kendra is a managed information retrieval and intelligent search service that uses natural language processing and advanced deep learning model.

AI Assistant