Can Screpy crawl a JavaScript site?

Its project settings include an Execute JavaScript option. Screpy tells users of JavaScript frameworks to enable it when the initial HTML is not sufficient for analysis.

Will every Screpy request contain the token `screpy`?

Not necessarily. The recorded crawler uses that token, but project owners can select or customize the request user agent.

Does Screpy respect robots.txt?

The available record does not confirm it. Publish a specific rule if desired, watch the result, and use access controls for a guaranteed block.

Does Screpy use crawled pages for AI answers or training?

Screpy applies the crawl to a customer's SEO audit. Its documentation does not connect that data to public AI answers or training, even though other parts of the platform use AI or report AI bot activity.

Screpy & AI Search: What to Know

What is Screpy?

Screpy crawls a website after a user adds that domain or URL as a project. The resulting private audit reports page status, redirects, canonical tags, internal links, crawl paths, and other site issues. It is not a web-wide indexing crawler.

Project settings determine the shape of the crawl. A user can restrict discovery to sitemap URLs, set a maximum page count, run an automatic or manual reanalysis, and enable JavaScript execution for sites that need rendering. Those choices explain why two Screpy crawls of the same site may request different resources.

This directory entry matches the short token screpy, but Screpy lets a project owner choose a default or custom user agent. Some authorized Screpy requests may therefore appear under another identity, while any unrelated client can copy this short token. Screpy's support material also tells customers to allow current crawler addresses when a firewall returns 403, but this bot does not mark IP verification as available.

Screpy describes its platform as AI-assisted and includes reports about third-party AI bot traffic. Those product features do not make the screpy crawler an AI search fetcher. Its documented job is to collect data for the customer's SEO workspace, with no stated model-training or public citation function.

Indirectly relevant

Is Screpy relevant for AI search?

Indirectly. Screpy has no AI product of its own, but its output can end up in the systems that AI answers draw on.

Screpy is an SEO auditing crawler. It has no direct effect on search rankings or AI answers, but the reports it feeds are used to fix crawlability and content issues that do affect how search and AI systems see your site.

How to handle Screpy

Allow Screpy when a known project owner needs the site audit. If the crawl receives 403 responses, use Screpy's current support documentation to review the firewall or CDN configuration instead of creating a broad exception for all bots.

To ask the recorded user agent not to crawl, add:

User-agent: screpy
Disallow: /

Robots.txt compliance is not confirmed for this entry. Check whether later screpy requests honor the rule. Apply a server or edge block when access must be denied, and remember that a Screpy user can configure a different request identity.

For an authorized project, reduce unnecessary load inside Screpy by choosing sitemap-only analysis or a suitable page limit. That is more precise than blocking the crawler after every audit starts.

Examples

A developer enables JavaScript execution for a React site, then sees Screpy request rendered pages during a manual reanalysis.
An agency limits a client audit to sitemap URLs so the crawler does not discover an unbounded faceted navigation space.
A security team denies unexplained `screpy` traffic because no project owner can identify an active audit.

Frequently asked questions about Screpy

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

A user adds a site to a Screpy project and starts an analysis, or a configured automatic reanalysis runs. The crawl is tied to that project rather than general web indexing.