What request rate does Coosti document?

The operator says it normally checks each product once per day and sends at most one request every two seconds to a domain. A site can ask for a longer pause with `Crawl-delay`.

Will CoostiBot honor robots.txt?

Yes. Both the crawler record and Coosti's bot page document robots.txt support. The operator specifically mentions `Disallow` and `Crawl-delay`.

How can a retailer verify CoostiBot traffic?

Match `CoostiBot/1.0` and compare the source address with the current list on Coosti's bot page. The published addresses are `140.99.60.199` and `168.93.216.108`, but that list can change.

Does CoostiBot contribute to AI search or training?

No such use is documented. The collected product data is described as input for Coosti's price comparison service.

CoostiBot & AI Search: What to Know

What is CoostiBot?

Coosti.dk uses CoostiBot to collect product information and prices from online retailers for its Danish comparison service. Its operator limits the crawl to public product pages. Prices and availability are the named data points, while account pages and checkout flows are outside the stated scope.

A product is typically checked once a day. Coosti also sets a ceiling of one request every two seconds for each domain and honors a longer Crawl-delay set by the site. The bot avoids pages marked noindex as well as paths excluded in robots.txt.

Requests identify themselves as Mozilla/5.0 (compatible; CoostiBot/1.0; +https://coosti.dk/bot-info). Coosti currently publishes 140.99.60.199 and 168.93.216.108 as its crawler addresses. Those addresses can help with verification, but site owners should recheck the operator page before maintaining a permanent allowlist.

CoostiBot feeds a price comparison catalog. It has no documented part in AI search or language-model training. Allowing it can keep a retailer's Coosti listings current; blocking it affects that shopping channel rather than generative search visibility.

Indirectly relevant

Is CoostiBot relevant for AI search?

Indirectly. CoostiBot has no AI product of its own, but its output can end up in the systems that AI answers draw on.

CoostiBot collects content from many sites and redistributes it through its own product. Aggregated copies can end up in datasets that AI systems later learn from, and some aggregators are themselves sources that AI answer engines draw on.

How to handle CoostiBot

Keep canonical product pages available if the store wants its offers refreshed on Coosti. When server load is the concern, set a crawl delay for this bot or exclude costly non-product routes before opting out of the service altogether.

Coosti documents support for this rule:

User-agent: CoostiBot
Disallow: /

A complete disallow can leave prices and availability stale on Coosti. Continue to protect customer and checkout areas with normal access controls rather than relying on crawler policy. After a robots change, check logs for the documented user agent and compare the source with Coosti's current IP list.

Examples

A furniture retailer wants to remain listed on Coosti but sees catalog rendering slow down, so it adds `Crawl-delay: 8` only for `CoostiBot` and leaves product pages open.
A merchant ending its Coosti participation disallows the named crawler, then checks whether its old offers stop receiving price and availability updates.

Frequently asked questions about CoostiBot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Coosti says it crawls publicly accessible product pages for product information, prices, and availability. It says it avoids protected areas, personal data, and the checkout process.