Does Amazon Product Discovery respect robots.txt?

Yes. AmazonProductDiscoverybot honors user-agent and Disallow rules. Changes may take up to 24 hours, and Amazon treats an unreadable robots file as if it does not exist.

Why are the request and robots names different?

Amazon documents `AmazonProductDiscovery/1.0` inside the HTTP user agent, but addresses robots rules to `AmazonProductDiscoverybot`. Use each value for its stated purpose.

Can an Amazon Product Discovery request be authenticated?

Amazon says its requests include Signature, Signature-Input, and Signature-Agent headers. Follow the public-key directory in Signature-Agent to validate the message rather than trusting the user agent alone.

Is this crawler used for AI search or model training?

Amazon's documentation assigns it a product-information role. It does not describe this crawler as collecting LLM training data or fetching sources for generated answers.

Amazon Product Discovery AI Crawler Guide

What is Amazon Product Discovery?

Amazon Product Discovery collects public product details from Amazon selling-partner, brand, and retailer websites. Amazon uses those details to improve the accuracy and completeness of product information available in its store. The intended downstream experience is product research and shopping on Amazon.

AmazonProductDiscoverybot honors User-agent and Disallow rules. Amazon says rule changes can take up to 24 hours to reach its systems, and an unreadable robots file is treated as though no file exists. The crawler does not support crawl-delay, nofollow, or noindex, so those directives cannot substitute for a targeted allow or disallow policy.

The crawler can request product pages that a merchant already exposes on the public web. It is distinct from AmazonSellerInitiatedListing, the related workflow where a seller supplies a URL to help create a product page in Amazon's store. Amazon documents both clients together, but gives them separate request identities.

The practical content consequence is specific to Amazon's product records. Allowing the crawler can let Amazon collect updated public product facts; blocking it prevents that client from collecting the disallowed paths. Amazon does not document AmazonProductDiscoverybot as an LLM training crawler or an AI answer fetcher, so a visit is not evidence of AI training, and a robots block is not a general Amazon AI opt-out.

Amazon's Product Bot documentation says live discovery requests include AmazonProductDiscovery/1.0 inside a browser-style user agent. The robots group uses a different documented name, AmazonProductDiscoverybot. This page has no catalog token, so those two operator-published values are the useful strings for request detection and crawl policy.

Amazon says every discovery request carries Signature, Signature-Input, and Signature-Agent HTTP headers. A verifier can fetch the public key directory named by Signature-Agent and validate the message signature. That is stronger evidence than the user agent alone, although this directory retains its unverifiable verification status and is not marked as IP verified.

Relevant for AI searchAmazon

Is Amazon Product Discovery relevant for AI search?

Yes. Amazon Product Discovery feeds Amazon, so the pages it can reach shape what those AI products say about you.

Amazon Product Discovery feeds a search index that also powers that engine's AI answer features and overviews. One crawl can serve a classic results page and a generated summary, so blocking it costs you both the rankings you would expect and a growing share of AI answers built on the same index.

How to handle Amazon Product Discovery

Allow the crawler on public catalog pages if you want Amazon to collect product details from them. Disallow internal search results, account areas, or other paths that should not participate. Authentication remains necessary for private data, especially because Amazon treats a robots file it cannot fetch as absent.

Amazon documents this exact robots identity:

User-agent: AmazonProductDiscoverybot
Disallow: /

Allow up to 24 hours for a changed rule to propagate. Do not use crawl-delay, nofollow, or noindex for this bot because Amazon says it does not support them. When checking a suspicious request, validate its HTTP Message Signature through the directory in Signature-Agent instead of trusting AmazonProductDiscovery/1.0 by itself.

Examples

A retailer updates the dimensions and material on a public product page. A later AmazonProductDiscovery request can collect those public details for Amazon's product information.
A merchant allows `/products/` but disallows an internal account path for AmazonProductDiscoverybot, then waits for the documented policy-update window.
A security team receives a browser-like request containing AmazonProductDiscovery/1.0 and validates the signed headers before treating it as Amazon traffic.

Frequently asked questions about Amazon Product Discovery

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Amazon says it collects publicly available product details from selling-partner, brand, and retailer websites to improve product information on Amazon.

Related bots

Amazon Seller Initiated Listing

Amazon's web crawler that helps sellers succeed by giving them the option to provide a URL to a website and create high-quality product pages in Amazon's store.

E-commerce