Can Cookiebot reach pages behind a login?

Cookiebot can configure a scan behind a login or cookie wall when the customer provides the login URL and credentials through its support process.

Why does one scan generate so many asset requests?

The scanner visits accessible pages with browser-like interactions and reloads resources without a cache. Cookiebot says the full scan commonly runs for many hours.

Will a Cookiebot robots.txt group stop the scanner?

It should not be treated as a dependable control. The supplied Cloudflare record says the bot does not follow robots.txt, while Cookiebot points customers to scanner filters and network controls.

Does blocking Cookiebot reduce AI visibility?

No. Cookiebot is not documented as an AI search or training collector. Blocking it affects the customer's cookie scan, not whether an AI service discovers or cites the site.

Cookiebot: What It Is & How to Handle It

What is Cookiebot?

A Cookiebot scan is an audit of the cookies and tracking technologies that visitors can encounter on a customer's site. Cookiebot uses the findings in its scan report and cookie declaration, which in turn support the consent information shown by the customer's banner.

Rather than reading a few static URLs, the scanner behaves much like a group of visitors. It follows navigable links, opens menus, scrolls, plays embedded media, and can place products in a cart. It does not submit forms or finish a purchase. Cookiebot support can configure access behind a login or cookie wall when the customer supplies the required details.

The requests use a browser-style header containing Cookiebot/1.0. Cookiebot says a scan may take up to 24 hours and does not cache resources, so one scan can produce a conspicuous run of page and asset requests. The operator publishes its scanner addresses but recommends matching the stable user-agent marker for allowlisting and analytics filtering.

Nothing in Cookiebot's stated purpose connects this traffic to AI search or model training. Its crawler is collecting a cookie inventory for the site owner, not building an answer index. Refusing the scan can make that inventory incomplete, but it does not keep the site's content out of an AI system.

Not relevant for AI search

Is Cookiebot relevant for AI search?

No. Cookiebot is not part of AI search or training, so allowing or blocking it does not change your AI visibility.

Cookiebot runs synthetic monitoring or uptime checks on a schedule. It is not collecting content for any index or model, so it has no bearing on search rankings or AI answers. You usually see it because your own team or a service you use set up a check.

How to handle Cookiebot

When the scan belongs to your organization, allow the published scanner addresses or match Cookiebot/1.0, then remove those visits from audience reporting if necessary. Cookiebot can sometimes filter repetitive dynamic URLs for a customer. Review the downloaded URL list and compare scans before and after any filter because an overly broad filter can hide trackers.

The recorded token can be addressed in robots.txt:

User-agent: Cookiebot
Disallow: /

Do not depend on that file for enforcement. The Cloudflare record supplied for this bot marks robots.txt support as false, and Cookiebot documents filters, user-agent matching, and firewall allowlisting instead of promising robots compliance. Block at the CDN or origin when access must stop, with the understanding that the resulting cookie report may be incomplete.

Examples

After a publisher adds a video player to its article template, Cookiebot opens an article, starts the embed, and records tracking activity that was absent from the previous report.
A site produces thousands of parameterized listing URLs during a scan. The owner reviews Cookiebot's URL export, requests a narrow filter for the duplicate pattern, and checks the next report for missing trackers.

Frequently asked questions about Cookiebot

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

It records cookies and other tracking technologies encountered while visiting the site's HTML pages and interacting with available content. Those findings go into the customer's scan report and cookie declaration.