Promptwatch Logo

Amazon Kendra

Amazon Kendra is a managed information retrieval and intelligent search service that uses natural language processing and advanced deep learning model.
AI AssistantSearch Engine Crawler

What is Amazon Kendra?

Amazon Kendra is AWS's managed search and information-retrieval service. Organizations use it to build search over documents they choose, with natural-language queries and answers drawn from the indexed material. It is an enterprise search product, not a public assistant that independently surveys the web.

Kendra's Web Crawler is one way a customer can feed pages into that index. The customer supplies seed URLs or sitemaps and chooses the crawl scope. AWS supports public and internal HTTPS sites, along with authenticated access and web proxies for some connector configurations.

AWS requires customers to crawl only pages they own or are authorized to index. That matters when this user agent appears on an unrelated public site: the request reflects one customer's Kendra data source, not an Amazon-wide collection program. The site may be destined for an internal knowledge search used by that customer's staff or users.

A successful crawl can make a page searchable inside the configured Kendra index. It does not place the page in a general Amazon search engine, ChatGPT, or another public AI answer service. AWS documentation describes the workflow as document indexing and retrieval, not collection for training a generative foundation model.

Cloudflare identifies Amazon as the operator and links to the Amazon Kendra documentation. Its directory records user agents such as amazon-kendra-customer-id-[id] and amazon-kendra-web-crawler-*, with the shared pattern amazon-kendra-. The customer-specific portion can vary between requests.

There is no single stable user-agent token stored for this page, and no Web Bot Auth key directory is listed. Treat the user agent as a classification clue, not proof that Amazon or a particular Kendra customer authorized the crawl. The requested host, path selection, timing, and responses are useful when investigating who configured the data source.

Relevant for AI searchAmazon Q

Is Amazon Kendra relevant for AI search?

Yes. Amazon Kendra feeds Amazon Q, so the pages it can reach shape what those AI products say about you.

Amazon Kendra crawls and indexes pages so the AI search or assistant behind it can retrieve them at answer time. A page it has never fetched cannot be quoted, summarized, or linked in that product's answers, so most sites keep it allowed to stay citable. Blocking it removes your pages from that AI surface and hands those citations to competitors.

How to handle Amazon Kendra

AWS says the Amazon Kendra Web Crawler respects standard Allow and Disallow directives in robots.txt. The supplied Cloudflare directory record marks its own followsRobotsTxt field as false, so use AWS's product documentation for the stated behavior and verify enforcement in your access logs.

This page does not store a stable user-agent token, so do not publish a rule built from the display name or a guessed AmazonKendra value. AWS documents the current robots syntax separately, while Cloudflare shows customer-specific user-agent forms. Check the exact Kendra connector version and AWS documentation before adding a targeted group.

Allow a crawl when you know the Kendra customer is authorized to index the material and want the pages available in that search deployment. For an unexpected request, ask the customer or system owner to identify the data source. A WAF block remains appropriate for sensitive paths or an unauthorized crawl, regardless of robots.txt.

Examples

  • A company configures its employee handbook as a Kendra web data source, and the crawler indexes the HTTPS pages for internal natural-language search.
  • A documentation team sees an `amazon-kendra-customer-id-` user agent and traces it to a search deployment created by another business unit.
  • A publisher receives an unexpected Kendra crawl, checks that robots.txt is being honored, and asks the requesting customer to confirm authorization.

Frequently asked questions about Amazon Kendra

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Amazon Web Services provides Kendra. Individual AWS customers configure the web crawler and choose which authorized sites become data sources.

Related bots

Amazon Q

Amazon Q Business is a generative artificial intelligence (generative AI)-powered assistant that you can tailor to your business needs.

AI Assistant

ChatGPT-Operator

Handles user-initiated requests from ChatGPT operator accessing external content; not used for automated crawling or AI training.

AI Assistant

ChatGPT-User

Handles user-initiated requests in ChatGPT, accessing external content to provide real-time information; not used for automated crawling or AI training.

AI Assistant

Claude-SearchBot

Claude-SearchBot is Anthropic's crawler for improving search results shown to Claude users.

AI Assistant

Claude-User

Claude-User supports Claude AI users. When individuals ask questions to Claude, it may access websites using a Claude-User agent.

AI Assistant

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.

Promptwatch Dashboard