Does Spyglasses respect robots.txt?

The stored record marks compliance as unconfirmed. Spyglasses currently states that its crawler follows bot-specific and wildcard directives, so test that stated behavior against requests to your site.

Does SpyglassesBot crawl the open web continuously?

According to Spyglasses, no. It fetches pages when a user requests an analysis of a registered or monitored site.

How can I verify Spyglasses is genuine?

Validate the request's Web Bot Auth headers with the public key directory at https://www.spyglasses.io/.well-known/http-message-signatures-directory. The user-agent text by itself is easy to spoof.

Does SpyglassesBot store or index my HTML?

The operator says it analyzes fetched content, stores the resulting analysis, and does not retain the raw HTML or build a search index.

Can a SpyglassesBot visit improve my AI rankings?

The visit can produce a report for a Spyglasses user, but it is not a submission to an AI answer engine. No source says that allowing this bot changes rankings, citations, or model training.

Spyglasses & AI Search: What to Know

What is Spyglasses?

SpyglassesBot fetches public pages for analyses requested by Spyglasses users. Depending on the job, it may inspect a sample of pages, read a sitemap, check AI crawler access, resolve a cited URL through redirects, or collect page content for a readiness audit.

Its user-agent suffix can describe the immediate task. The recorded variants include AI Visibility Report, Brand Consistency Checker, FAQ Generation, Page Title Fetcher, and Citation Resolver. All of them contain SpyglassesBot, which is the common marker to use in log filters.

Spyglasses says these crawls run on demand rather than continuously. Its documentation also says it does not build a search index and does not retain raw HTML after analysis. Requests can carry Web Bot Auth signatures, which lets an edge service verify the sender without trusting a user-agent string alone.

The product studies how pages are presented to AI assistants, but SpyglassesBot is not itself a public AI search crawler. Allowing it lets a Spyglasses customer complete an audit. It does not make the page available to ChatGPT or another answer engine, and its documentation does not state that fetched HTML is used for model training.

Indirectly relevant

Is Spyglasses relevant for AI search?

Indirectly. Spyglasses has no AI product of its own, but its output can end up in the systems that AI answers draw on.

Spyglasses is an SEO auditing crawler. It has no direct effect on search rankings or AI answers, but the reports it feeds are used to fix crawlability and content issues that do affect how search and AI systems see your site.

How to handle Spyglasses

Allow SpyglassesBot when you or an authorized monitoring partner uses Spyglasses. If an audit should cover only public sections, apply a bot-specific rule and confirm that the requested analysis still has the pages it needs:

User-agent: SpyglassesBot
Disallow: /account/
Disallow: /internal/

The data record leaves robots.txt compliance unconfirmed, while the current operator documentation says SpyglassesBot honors wildcard and bot-specific rules, including Allow, Disallow, and Crawl-delay. Treat that as the operator's stated policy and verify it in logs.

For authenticated bot handling, validate Web Bot Auth signatures against https://www.spyglasses.io/.well-known/http-message-signatures-directory. A bare SpyglassesBot header can be copied by anyone and is not proof of origin.

Examples

A site owner starts an AI readiness audit. The logs show several `SpyglassesBot` requests whose suffix names the audit, followed by a sitemap request used to select pages.
A monitoring team resolves a URL cited in an assistant response. One `Citation Resolver` request follows the redirect chain to the final page without starting a site-wide crawl.

Frequently asked questions about Spyglasses

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

The ending labels the task that caused the request, such as an AI visibility report or citation resolution. Match `SpyglassesBot` when you want one rule or log filter to cover every recorded variant.