Does Amazon Bedrock AgentCore Browser respect robots.txt?

Its robots behavior is unknown, and no stable token is recorded. Do not assume that rules for Amazon's separate `bedrockbot` crawler apply to AgentCore Browser.

Does AgentCore Browser always sign its requests?

No. AWS says Web Bot Auth is disabled by default and must be enabled on a custom browser. When enabled, the service adds HTTP message-signature headers to each request.

How can I verify Amazon Bedrock AgentCore Browser is genuine?

For a signing-enabled browser, validate the HTTP message signature with the public-key directory identified by the `Signature-Agent` header. The facts record does not provide a fixed user-agent token or Cloudflare identity entry.

Does AgentCore Browser affect AI search or model training?

The browser service has no direct search-indexing or training role. What happens to content depends on the developer's agent application and the task that opened the browser session.

Is AgentCore Browser the same as bedrockbot?

No. `bedrockbot` is associated with Amazon Bedrock's web crawler for knowledge bases. AgentCore Browser is an interactive browser environment for agents.

Amazon Bedrock AgentCore Browser Bot Guide

What is Amazon Bedrock AgentCore Browser?

Amazon Bedrock AgentCore Browser is a managed, cloud-based browser that developers can give to an AI agent. It is infrastructure for running browser sessions, not one Amazon assistant that independently decides which sites to visit.

AWS documents an isolated environment where an agent can navigate web applications, click controls, fill forms, take screenshots, and extract information. The developer can use the AWS-managed browser or create a custom browser with its own network and recording settings. The particular application and user instruction determine why a website receives traffic.

A session can contain many requests because the browser works through an interactive flow. AWS provides live viewing, CloudWatch metrics, and session replay or recording options for observing that work. From the website's side, requests may look more like an ordinary browser journey than a crawler repeatedly downloading documents.

AgentCore Browser does not by itself index pages for AI search or collect a model training dataset. Any content it reads is used according to the agent application that opened the session. It is also separate from Amazon Bedrock's web crawler for knowledge bases, which uses bedrockbot identifiers and publishes its own robots.txt behavior.

The facts record contains no stable user-agent token and no Cloudflare catalog entry for AgentCore Browser. AWS does offer Web Bot Auth, but browser signing is disabled by default and must be enabled when a custom browser is created. When enabled, AgentCore signs HTTP requests and sends Signature, Signature-Agent, and Signature-Input headers for verification.

No robots.txt commitment is recorded for AgentCore Browser. A signed request proves the identity represented by the signature; it does not require a site to allow that request. AWS notes that site owners may still block, monitor, or rate-limit signed traffic, and not every bot-control provider or website will recognize it.

Relevant for AI searchAmazon Bedrock

Is Amazon Bedrock AgentCore Browser relevant for AI search?

Yes. Amazon Bedrock AgentCore Browser feeds Amazon Bedrock, so the pages it can reach shape what those AI products say about you.

Amazon Bedrock AgentCore Browser crawls and indexes pages so the AI search or assistant behind it can retrieve them at answer time. A page it has never fetched cannot be quoted, summarized, or linked in that product's answers, so most sites keep it allowed to stay citable. Blocking it removes your pages from that AI surface and hands those citations to competitors.

How to handle Amazon Bedrock AgentCore Browser

Do not add an AgentCore Browser robots.txt group. There is no stable token in the record, and the browser's robots behavior is unknown. A guessed token could create a false sense of control or interfere with the separate bedrockbot crawler.

Decide access according to the browser action. Public reading may be acceptable while account changes, payments, scraping-sensitive routes, or high-volume automation require authentication, confirmation, rate limits, or a block. Apply those checks to every browser client, not only requests you recognize as AgentCore.

If your policy allows only identified agents, require Web Bot Auth and validate the HTTP message signature through the directory named by Signature-Agent. Signing is optional and disabled by default, so an unsigned request cannot be reliably attributed to AgentCore Browser from this bot. A correctly signed request can still be denied under your site's policy.

Keep this handling separate from Amazon Bedrock knowledge-base crawling. Rules for bedrockbot do not establish what an AgentCore Browser session may do.

Examples

A support agent uses an isolated AgentCore Browser session to reproduce a customer's web workflow while an operator watches through Live View.
A site accepts a signed AgentCore session on public help pages but still requires a normal login before the agent can change an account.
A security team keeps its `bedrockbot` knowledge-base rule separate from policy for interactive AgentCore Browser traffic.

Frequently asked questions about Amazon Bedrock AgentCore Browser

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Amazon provides the AgentCore Browser service. The developer operating the agent application chooses the websites, tasks, and session configuration.

Related bots

Amazon Product Discovery

Amazon Product Discovery collects public product details from Amazon selling-partner, brand, and retailer websites.

Search Engine Crawler

Amazon Seller Initiated Listing

Amazon's web crawler that helps sellers succeed by giving them the option to provide a URL to a website and create high-quality product pages in Amazon's store.

E-commerce

Amazon Bedrock Bot

Amazon Bedrock Bot runs when an Amazon Bedrock customer configures a website as a data source for a knowledge base.

AI Crawler

Amzn-SearchBot

Amzn-SearchBot crawls for improving Amazon search experiences (Alexa, Rufus).

AI Search

Amazon Kendra

Amazon Kendra is a managed information retrieval and intelligent search service that uses natural language processing and advanced deep learning model.

AI Assistant

Amazon Bedrock AgentCore Browser