What is ICC Crawler?
ICC-Crawler is a web collection program run by the Universal Communication Research Institute at Japan's National Institute of Information and Communications Technology, or NICT. Its official crawler page says it automatically travels the Internet and collects web pages.
NICT's current policy says the crawler reads the robots.txt file on each target host and follows its access restrictions. When a site sets Crawl-Delay, ICC-Crawler uses either that interval or its own minimum interval, whichever is longer. This gives operators both scope and pacing controls.
An archived NICT description ties the collection to research on web search and data mining. The available material does not say that pages train a foundation model or feed a public AI answer service. An ICC-Crawler request should therefore be treated as research collection, not as proof of model training, AI search eligibility, or a future citation.
Cloudflare's bot directory records the user agent as ICC-Crawler/3.0 (Mozilla-compatible; ; https://ucri.nict.go.jp/en/icccrawler.html). The stable robots.txt token is ICC-Crawler. Versioned log matching should use the token rather than requiring the complete string.
The bot is verified in the directory, but this entry has no verified IP ranges or HTTP message-signature directory. The user-agent header still can be copied by another client. If an exception would bypass a security control, origin identity needs evidence beyond the string.
Sites that support NICT's web research can allow public pages while excluding expensive endpoints, personal-information routes, or file classes. NICT also invites operators to contact the institute if collection continues after a robots.txt restriction, giving sites a second route when the published rule does not produce the expected result.
