How often does the crawler visit?

NewsNow says it aims to scan a publisher's headline source every 10 minutes, but it explicitly says the interval can vary.

Why was a crawled article not listed?

A crawl is only the collection step. NewsNow may omit an article after topic analysis, and its duplicate system can suppress a headline that matches one already found.

Does NewsNow respect robots.txt?

Its publisher documentation tells site owners to remove rules that prohibit the `NewsNow` spider. Because this dataset does not independently confirm compliance, verify the result in access logs.

Is NewsNow an AI training crawler?

No training role is documented. NewsNow describes article analysis and feed placement for its aggregation service, which is not evidence of language-model training.

NewsNow: AI Crawler Guide & robots.txt

What is NewsNow?

NewsNow is a news aggregation crawler, not a general web search spider. It visits participating publishers so NewsNow can display links to recent reporting in its topic feeds.

Publishers give NewsNow a dedicated headline source, usually an RSS feed, an Atom feed, or a simple page containing headline links. NewsNow says it aims to scan that source every 10 minutes, then requests new articles and analyzes their text to decide which feeds fit. The timing and inclusion of any article are not guaranteed.

That workflow produces a recognizable access pattern: repeated requests to the designated feed followed by requests to newly linked stories. NewsNow may reproduce a headline and URL, sometimes with a short excerpt. It also removes duplicate headlines, so a successful crawl does not mean every story will appear.

The article analysis described by NewsNow is automated classification for a news service. The available material does not say that the crawler trains a language model or supplies a general AI answer engine. Its direct visibility effect is on NewsNow's own news feeds.

Relevant for AI search

Is NewsNow relevant for AI search?

Yes. NewsNow collects pages for an AI product, so what it can crawl influences how AI systems describe your brand.

NewsNow feeds a search index that also powers that engine's AI answer features and overviews. One crawl can serve a classic results page and a generated summary, so blocking it costs you both the rankings you would expect and a growing share of AI answers built on the same index.

How to handle NewsNow

If your publication participates in NewsNow, allow the bot to reach the submitted headline source and linked articles. Keep that source current, use a distinct URL and headline for each story, and check the feed first when a headline is missing. A non-publisher seeing this traffic should confirm whether another team submitted the site.

To exclude NewsNow from the whole site, use its stable token:

User-agent: NewsNow
Disallow: /

NewsNow's publisher guidance explicitly checks whether robots.txt prohibits its spider and shows publishers how to allow it. The bot record does not contain a separate verified compliance flag, so confirm any allow or disallow change in your logs.

Examples

A newsroom adds a new story to its NewsNow RSS feed and sees the bot fetch the feed before requesting the article URL.
An editor investigates a missing headline and discovers that the same headline was already published elsewhere, triggering NewsNow's duplicate handling.
A publisher retiring its NewsNow account blocks the `NewsNow` token and watches for any remaining requests to the old headline page.

Frequently asked questions about NewsNow

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

For participating publishers, it scans a dedicated headline page or feed and then requests the linked articles so it can classify them into news topics.