Does Grok Search respect robots.txt?

Its status is partial. Target `xAI-Grok` in robots.txt, then verify actual requests in your logs. Use server or edge enforcement when a disallow must be guaranteed.

Will allowing Grok Search guarantee a citation?

No. Allowing access only makes the page available for retrieval. Grok decides which sources fit a query and may cite another page or answer without a web citation.

Is Grok Search the same as Grok DeepSearch?

No. Grok Search handles live page retrieval for answers and uses `xAI-Grok`. DeepSearch is a multi-step research workflow with the separate `Grok-DeepSearch` token.

Is `xAI-Grok` proof that a request came from xAI?

No. The token is easy to copy, and no signature keys or confirmed xAI network ranges accompany this bot. It is a log label, not a credential.

Grok Search: AI Crawler Guide & robots.txt

What is Grok Search?

Grok Search is xAI's live web fetcher for search-backed answers in X and the Grok apps. It uses the token xAI-Grok, which is different from the Grok-DeepSearch identity used for longer research workflows.

When a Grok answer needs current web information, the search system can find a page, open it, and extract material relevant to the question. xAI's Web Search documentation describes real-time search and page browsing, with citations returned alongside the generated response.

For publishers, a successful fetch makes the page available as a current source for that answer. It may lead to a quote or link, but selection depends on the query and other sources. Blocking the bot can stop this retrieval path without removing facts that Grok already knows or can obtain elsewhere.

Grok Search affects live AI search, not documented model training. The bot record places it in the AI Assistant and Search Engine Crawler categories, and xAI describes the associated workflow as web search. No source supplied for this page says that xAI-Grok requests are used to build a training corpus.

Robots.txt handling is partial. A group for xAI-Grok can communicate an allow or disallow preference, but site owners should not assume every live fetch will honor it. Rate limits, authentication, and edge access controls remain the reliable ways to protect content that must not be retrieved.

The only identity marker supplied here is the user-agent token. There is no corresponding Cloudflare record, signature key directory, or confirmed xAI address list, so Grok Search is marked unverifiable. Use xAI-Grok to group routine log entries, not to authenticate a client.

Relevant for AI searchGrok

Is Grok Search relevant for AI search?

Yes. Grok Search feeds Grok, so the pages it can reach shape what those AI products say about you.

Grok Search crawls and indexes pages so the AI search or assistant behind it can retrieve them at answer time. A page it has never fetched cannot be quoted, summarized, or linked in that product's answers, so most sites keep it allowed to stay citable. Blocking it removes your pages from that AI surface and hands those citations to competitors.

Track Grok Search with Promptwatch

Promptwatch classifies Grok Search (xAI-Grok) in real time from your server and CDN logs. See exactly when it visits, which pages it requests, the status codes it gets, and how those crawls map to AI citations.

How to handle Grok Search

Allow Grok Search on public pages that you are comfortable having retrieved for current Grok answers. Block it if that use is unwanted, but keep private or licensed material behind real authorization controls regardless of the bot rule.

The robots.txt group uses the exact stable token:

User-agent: xAI-Grok
Disallow: /

Because compliance is partial, review subsequent requests and enforce the decision at your edge if necessary. A 401 or 403 response, or an authenticated application boundary, is stronger than user-agent filtering.

Treat xAI-Grok as a classification hint only. Do not exempt the request from rate limits or security checks because the identity has no supplied signature or verified IP range.

Examples

A news site publishes a correction, then sees `xAI-Grok` retrieve the updated article while Grok answers a question about the developing story.
A documentation team leaves release notes public so Grok Search can use the latest version rather than an older model memory.
A members-only database returns an authorization challenge to Grok Search instead of relying on its partially honored robots.txt group.

Frequently asked questions about Grok Search

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

xAI uses it for real-time web retrieval behind Grok search and answer features in X and the Grok apps. xAI also exposes web search as a tool in its developer API.