All Africa Crawler

All Africa Crawler is part of AllAfrica Global Media's news aggregation and distribution operation. AllAfrica works with more than 140 news organizations and over 500 other institutions and individuals, collecting reporting from across Africa for republication and delivery through its own channels.

The crawler record describes AllAfrica NewsBot/1.0 fetching content from publishers that have written agreements with AllAfrica, as well as material available without licensing restrictions or otherwise free to distribute. AllAfrica says the original author and institution are credited. This is a content-ingestion workflow, not a general-purpose web search crawl.

Requests carry the stable AllAfrica token, but the available record does not establish that the crawler honors robots.txt. A robots rule is still useful for stating a preference, though publishers with licensing or access concerns should verify the result in logs and use enforceable controls where needed.

Nothing in the matching record says the collected articles train an AI model or feed an AI answer engine. Wider syndication may expose a story to more readers and downstream services, but that is not evidence of AI search inclusion or training reuse.