Does GrokBot respect robots.txt?

Only partially, according to the recorded status. A specific rule is still useful, but logs or edge enforcement are needed when the restriction must be dependable.

Is GrokBot the crawler behind every live Grok web answer?

That is not documented. xAI offers a real-time Web Search tool, but its public material does not connect every answer-time fetch to the `GrokBot` token.

Is this the same as the current Grok Bot agent?

No. The current agent product is documented separately, and the supplied directory record gives that agent the user agent `GrokAgent`. This page covers the `GrokBot` training crawler token.

Does a GrokBot request prove my content trained Grok?

No. It shows that a client using the token requested the URL. It does not reveal retention, filtering, dataset membership, or use in a model training run.

Can I verify GrokBot by IP address or signature?

No verified IP list or request-signing directory is recorded for this crawler. Its user-agent string can be spoofed, which is why the entry's verification status is `unverifiable`.

GrokBot: AI Crawler Guide & robots.txt

What is GrokBot?

GrokBot is the GrokBot crawler entry attributed to xAI for gathering public web material used in training the Grok model family. xAI has published little crawler-specific documentation for this token. The operator and stated purpose are known, but the entry remains marked unverifiable because there is no recorded technical method for proving that a request came from xAI.

The name now needs care. xAI also documents a live Web Search tool for Grok that can browse pages when answering a query, and its current Grok Bot product uses persistent agents that operate websites and apps. Those products do not establish that a request carrying GrokBot is a live search fetch or an interactive agent action. The supplied agent directory identifies the newer agent with GrokAgent, a different token.

For this crawler, the documented implication is training data collection. A fetched page may contribute material to a future Grok model, but a single request cannot show whether xAI retained the content or used it in a training run. It also does not guarantee that Grok will mention, recommend, or cite the page in an answer.

That difference separates model development from real-time retrieval. A site may allow live answer-time access while declining AI training, but xAI has not published a complete token map that would let GrokBot represent both choices. Rules and log analysis should therefore stay tied to the exact user agent observed.

GrokBot is classified as only partially respecting robots.txt. A group for GrokBot still records the site's preference and may stop compliant crawling, but it should not be treated as a guaranteed access control. The available facts do not explain which requests honor rules or why behavior is partial.

There is no published IP range or Web Bot Auth directory for this crawler in the source record. Anyone can copy a user-agent string, so a GrokBot log match is a classification signal rather than cryptographic identity. If exclusion is mandatory, combine robots.txt with an edge or server policy and review the resulting requests.

Relevant for AI searchGrok

Is GrokBot relevant for AI search?

Yes. GrokBot feeds Grok, so the pages it can reach shape what those AI products say about you.

GrokBot gathers public web content that can end up in the training data for large language models. Once your pages are in that set, they influence how the operator's models talk about you for that model generation. Allowing it lets your own writing carry weight in those answers; blocking it means the models learn about you from third parties instead.

Track GrokBot with Promptwatch

Promptwatch classifies GrokBot (GrokBot) in real time from your server and CDN logs. See exactly when it visits, which pages it requests, the status codes it gets, and how those crawls map to AI citations.

How to handle GrokBot

Use a specific rule if your policy excludes collection for model training. Do not describe the rule as a way to opt out of xAI's separate live search or agent products, since the available documentation does not map those products to this token.

User-agent: GrokBot
Disallow: /

Robots behavior for GrokBot is recorded as partial. Publish the rule, note when it became available, and check later requests to see how the crawler behaves on your hosts. For content that must not be fetched, enforce the decision at the application or edge rather than relying on robots.txt alone.

Examples

An independent publisher permits xAI's answer-time search traffic but blocks the separate `GrokBot` token because its licensing policy excludes model training.
A documentation team adds the robots rule, then checks whether requests to versioned manuals continue after the file is recrawled.
A security analyst sees a client claiming to be GrokBot and records it as unverified until other request evidence supports the attribution.

Frequently asked questions about GrokBot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

This entry identifies an xAI crawler used to collect web content for training the Grok family of models. xAI publishes limited documentation for the `GrokBot` token.