Promptwatch Logo

i-search-crawler

i-search-crawler is a web crawler operated by.
Scala Communications, Inc.i-search-crawler
Search Engine Crawler

What is i-search-crawler?

i-search-crawler belongs to Scala Communications' i-search service, a hosted search engine installed on a customer's own website. It builds the content index behind that site's search box rather than a public, web-wide search engine.

Scala documents scheduled automatic crawls and a manual crawl option for updates that need to reach search results sooner. Standard indexing covers HTML and PDF files. The service can also show page capture images and search across more than one configured domain.

The recorded user agent is i-search-crawler / 4.0. On a customer site, repeated requests should track the pages and documents selected for its internal search index. this bot does not publish cryptographic verification or an IP range, so the string still needs to be treated as a claim rather than authentication.

No public AI answer or model-training role is documented for this crawler. Blocking it affects the i-search index configured for that website, which can make its own search results stale or incomplete. It does not provide a general AI opt-out.

Relevant for AI search

Is i-search-crawler relevant for AI search?

Yes. i-search-crawler collects pages for an AI product, so what it can crawl influences how AI systems describe your brand.

i-search-crawler feeds a search index that also powers that engine's AI answer features and overviews. One crawl can serve a classic results page and a generated summary, so blocking it costs you both the rankings you would expect and a growing share of AI answers built on the same index.

How to handle i-search-crawler

Sites using i-search should allow the crawler only on content intended for their site-search users. Remove unwanted directories through the i-search management settings and keep confidential files behind authentication. After a large documentation update, a manual crawl can refresh the index sooner than the normal schedule.

To exclude the crawler from the site, use:

User-agent: i-search-crawler
Disallow: /

this bot marks robots.txt compliance as supported. Confirm the next crawl reflects your rule, especially when cached robots instructions or a manually started crawl may still be in progress.

Examples

  • A university updates a group of PDF policies and starts a manual i-search crawl so its site search returns the new versions.
  • A support site excludes an archive directory from i-search while continuing to index current HTML documentation.
  • An administrator who does not recognize `i-search-crawler / 4.0` checks whether another department installed Scala's hosted search before blocking it.

Frequently asked questions about i-search-crawler

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Scala Communications uses it to build the index for its hosted i-search site-search service.

Related bots

AdIdxBot

An AdIdxBot visit usually starts with a Bing Ads workflow rather than an organic search crawl.

Search Engine Crawler

Adsense

The AdSense crawler visits participating sites in order to provide them with relevant ads.

Search Engine Crawler

adsnaver

adsnaver is tied to paid search advertising, not Naver's general web index.

Search Engine Crawler

Algolia

The Algolia Crawler extracts content from your site and makes it searchable.

Search Engine Crawler

Amazon AdBot

Amazon AdBot scans pages that request ads from Amazon's advertising systems.

Search Engine Crawler

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.

Promptwatch Dashboard