How does the Mars Finder crawler identify itself?

The recorded user agent is browser-like and contains `(dbot)`. Use `dbot` as the stable token instead of matching a complete browser version string.

Does Mars Finder respect robots.txt?

The supplied records say yes. A specific `dbot` record can allow or disallow paths, but confidential content should remain protected by access controls.

Can Mars Finder search documents as well as HTML pages?

Its product page says site search can include PDF, Word, Excel, and PowerPoint files. Public documents in the configured scope may therefore receive crawler requests.

Does Mars Finder use crawled pages in AI answers?

Within a customer's Mars Finder installation, it can. The service advertises generated answers that summarize information from the site alongside regular results.

Does dbot train a general AI model?

No such use is documented in the supplied evidence. The stated AI workflow is retrieval and summarization for the enrolled site's own search experience.

Mars Finder: AI Crawler Guide & robots.txt

What is Mars Finder?

Mars Finder crawls for site search, not a web-wide consumer engine. Mars Flag provides the service to organizations that want visitors to search content across one or more registered sites. The resulting index belongs to that installed search experience.

Mars Finder now offers an on-site feature that generates an answer by aggregating and summarizing content from the customer's site alongside conventional results. This gives its crawl a real AI-answer impact, but only inside that Mars Finder installation. The available sources do not say dbot trains a general model or influences external assistants.

A customer places Mars Finder's search interface on its site, and the service returns results through an API. Its product documentation says the search can cover ordinary web pages and published documents such as PDF, Word, Excel, and PowerPoint files. The crawler supplies the content index that makes those results possible.

Recorded traffic uses a browser-shaped user agent ending in (dbot). Match the stable dbot token in logs and policy rules rather than the operating-system or browser version text around it. On an enrolled site, revisits are expected when the service needs to refresh its view of searchable content.

Relevant for AI search

Is Mars Finder relevant for AI search?

Yes. Mars Finder collects pages for an AI product, so what it can crawl influences how AI systems describe your brand.

Mars Finder feeds a search index that also powers that engine's AI answer features and overviews. One crawl can serve a classic results page and a generated summary, so blocking it costs you both the rankings you would expect and a growing share of AI answers built on the same index.

How to handle Mars Finder

Organizations using Mars Finder should let dbot reach the pages and files intended for site search, while excluding staging areas, administrative URLs, and documents that should not appear in results. Authentication is still necessary for confidential files, even when they are disallowed.

To remove the entire site from the crawler's scope, use:

User-agent: dbot
Disallow: /

Mars Finder's dbot is marked robots.txt-compliant in both this bot and Cloudflare's directory. Check a representative disallowed path after deployment, especially when an existing index may still contain an older copy. If you do not use Mars Finder but see sustained dbot traffic, ask Mars Flag or the organization managing the site's search configuration to confirm why the domain was enrolled.

Examples

A manufacturer publishes a revised PDF manual and confirms that `dbot` refetched it before testing the document in the site's Mars Finder search box.
A support team updates a troubleshooting page, then checks both the standard search result and the Mars Finder AI answer for stale instructions.
During a redesign, developers disallow the staging host and keep it behind authentication so draft pages cannot enter the production search index.

Frequently asked questions about Mars Finder

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Mars Flag provides it as a site-search service. The crawler indexes content from configured websites so visitors can search those sites through an embedded interface.