New Feature
Track Exa's Crawlers
Crawler Logs now tracks Exa, the search engine many AI agents and apps use to find and read web pages. You can see when Exa indexes your site and when an agent fetches one of your pages to answer a question.
- Exa Search Bot builds Exa's search index and counts as an AI crawl.
- Exa Citations is the live fetch Exa makes when a person or an app built on its API asks for a page, so it counts as a citation crawl.
- Both show up in the AI crawlers filter, the charts, and the logs table with Exa's icon. Exa publishes no IP ranges, so neither gets a Verified badge.
- The robots.txt generator for AI now writes an ExaSearchBot group whenever you add rules for search engines. Without one, ExaSearchBot follows your Googlebot and Bingbot rules instead of your default.
See which pages AI agents read
- Filter Crawler Logs to Exa Citations to find the pages agents fetch most. Those pages get quoted in answers, so keep their facts and prices current.
- If Exa Search Bot never visits a page you want agents to find, check that robots.txt doesn't block ExaSearchBot.
