What is KarlaCrawler?
KarlaCrawler belongs to Karla ApS, a Copenhagen company that makes AI customer service tools: Karla Chat, Karla Search and Karla Inbox. The Karla bot page says that when a client adds Karla's script to their site, the crawler fetches the client's sitemap.xml and reads its public pages. The content is indexed for retrieval-augmented generation, so Karla's assistant answers from the client's own, current information.
Karla says it only crawls websites where it has an active service agreement and explicit authorization from the domain owner. It also says the data is used only for the client's own service and is not used to train or develop the underlying AI models.
Requests use the user agent Mozilla/5.0 (compatible; KarlaCrawler/1.0; +https://getkarla.ai/en/bot) and carry an extra header, X-Karla-Crawler: Karla/1.0; contact=https://getkarla.ai/en/bot. Karla publishes a single source IP, 34.141.120.151, which you can allowlist in a firewall. Unlike many newer bots, it does not sign requests with Web Bot Auth.
Karla says the crawler strictly respects robots.txt. If you use Karla, a disallow rule also stops your chatbot's knowledge base from updating for the blocked paths.
