What user agent should I look for?

The recorded full value is `KeldanNewsCrawler/1.0`. Use the stable `KeldanNewsCrawler` token for log searches and robots rules.

Does KeldanNewsCrawler respect robots.txt?

The available records do not confirm compliance, and Cloudflare's bot directory marks it false. Treat a robots rule as a request, then verify behavior from signed traffic in your logs.

Does KeldanNewsCrawler collect content for AI training?

No published source in the supplied evidence says that it does. Its stated job is collecting Icelandic news for Keldan's own news search.

KeldanNewsCrawler AI Crawler Guide

Q: How can I verify KeldanNewsCrawler is genuine?

Validate its HTTP message signature with the public keys at `https://news-crawler.keldan.is/.well-known/http-message-signatures-directory`. A plain user-agent match is not proof because any HTTP client can copy it.

What is KeldanNewsCrawler?

KeldanNewsCrawler is aimed at Icelandic news, not the web at large. Keldan ehf. uses it to collect news articles for search on keldan.is, a site that presents Icelandic news alongside market information.

A fetched article may become discoverable through Keldan's news search. No supplied source says that KeldanNewsCrawler trains a language model or feeds a general AI answer engine. Its direct visibility effect is therefore limited to the Keldan search surface unless Keldan documents another use.

The recorded user agent is KeldanNewsCrawler/1.0. Publishers should expect article URL requests in their server or CDN logs. The available record does not explain how the crawler discovers stories, how often it revisits them, or whether it retains full text, so those details should be learned from observed traffic rather than assumed.

This crawler has a stronger verification option than most small search bots. Cloudflare's record includes a signed-agent directory at https://news-crawler.keldan.is/.well-known/http-message-signatures-directory. An edge that supports HTTP message signature verification can use those published keys to distinguish Keldan's requests from clients that merely copy the name.

Relevant for AI search

Is KeldanNewsCrawler relevant for AI search?

Yes. KeldanNewsCrawler collects pages for an AI product, so what it can crawl influences how AI systems describe your brand.

KeldanNewsCrawler feeds a search index that also powers that engine's AI answer features and overviews. One crawl can serve a classic results page and a generated summary, so blocking it costs you both the rankings you would expect and a growing share of AI answers built on the same index.

How to handle KeldanNewsCrawler

An Icelandic publisher that wants stories included in Keldan search can allow article pages while keeping subscriber accounts, preview endpoints, and editorial systems inaccessible. Authentication remains the control for nonpublic material; a robots rule is not an access boundary.

To request a complete crawl stop, use the recorded token:

User-agent: KeldanNewsCrawler
Disallow: /

Robots compliance is not established. The bot record leaves it unconfirmed, while Cloudflare's bot directory marks the crawler as not following robots.txt. Watch for signed requests to disallowed paths, and enforce a block at the edge if the rule is not honored. Signature verification confirms who sent a request, but it does not guarantee that the sender followed your crawl policy.

Examples

A Reykjavík newsroom checks whether KeldanNewsCrawler fetched a newly published business article after its canonical URL went live.
A CDN rule verifies the request signature before exempting KeldanNewsCrawler from a generic bot challenge.

Frequently asked questions about KeldanNewsCrawler

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

The documented scope is Icelandic news articles for the news search on keldan.is. The record does not describe a broader web or commercial search index.