How does the crawler discover pages?

Customers can start from a root URL, submit a sitemap, or combine both methods. The crawler follows the resulting configuration to build the index.

How should I exclude a page from Site Search 360?

Use the product's blacklist, whitelist, or no-index URL settings. Those controls define the hosted search index more precisely than a whole-site robots block.

Does SiteSearch360 honor robots.txt?

That behavior is not confirmed in the available bot record. Publish a rule if appropriate, then verify it in logs or enforce it at the server.

Is SiteSearch360 crawling for AI training?

No public-web training role is documented. The crawler's stated job is to populate search indexes for customer websites.

SiteSearch360: AI Crawler Guide & robots.txt

What is SiteSearch360?

SiteSearch360 crawls websites whose owners use the Site Search 360 hosted search box. A customer supplies a root URL, a sitemap, or both, and the service builds an index for searches performed on that customer's site.

During indexing, the crawler chooses page content for result titles, images, and descriptions. Customers can narrow the index with URL blacklists or whitelists, skip indexing for matching paths, and specify content with CSS selectors or XPath when automatic extraction picks the wrong section.

A SiteSearch360 indexing run uses the recorded user agent Mozilla/5.0 (compatible; SiteSearch360/1.0; +https://sitesearch360.com/). Crawls can include pages and linked documents selected by the customer's configuration. The record offers no signature endpoint or verified IP range, so a matching user agent is not enough to authenticate a request.

Site Search 360 advertises semantic search and AI-related product options, but its crawler is not documented as a public-web model-training bot. Blocking it usually damages the site's own hosted search index rather than changing visibility in outside AI answers.

Relevant for AI search

Is SiteSearch360 relevant for AI search?

Yes. SiteSearch360 collects pages for an AI product, so what it can crawl influences how AI systems describe your brand.

SiteSearch360 feeds a search index that also powers that engine's AI answer features and overviews. One crawl can serve a classic results page and a generated summary, so blocking it costs you both the rankings you would expect and a growing share of AI answers built on the same index.

How to handle SiteSearch360

For an active Site Search 360 account, control the index in its Data Sources and crawling settings first. Blacklist duplicate or irrelevant paths, use a sitemap for deliberate URL coverage, and keep private material inaccessible without authentication. Reindex after changing extraction or URL rules.

If you want to request a complete stop through robots.txt, use:

User-agent: SiteSearch360
Disallow: /

The available bot record does not confirm robots.txt compliance. Check the next crawl rather than assuming the rule was applied. Cancel the configured data source or add a server-side block when the crawler must not make requests.

Examples

A documentation team gives Site Search 360 its XML sitemap and checks the index for PDFs that are not linked from the homepage.
A retailer blacklists faceted category URLs so its on-site search contains canonical product pages instead of duplicates.
A site owner uses a CSS selector to index article text without navigation labels that were polluting search snippets.

Frequently asked questions about SiteSearch360

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

A Site Search 360 account has likely configured the domain as a data source for its hosted on-site search. Confirm the account owner before changing access.