CitibotSiteCrawler

CitibotSiteCrawler is Citibot's crawler for public information on government websites. The collected material supports Citibot's civic engagement tools, which government agencies use to answer resident questions through their own service channels.

Citibot has not published a robots.txt commitment for this identity. Cloudflare's bot directory marks the crawler as not following robots.txt, while the bot's listing leaves the status unknown. A robots rule can state the site's preference, but an agency that must prevent access should use a server, firewall, or authentication control as well.

Citibot describes its AI as a closed system that retrieves information from an agency's approved knowledge base rather than searching the open web for every answer. In that setup, the crawler supplies source material from public government pages so the deployed service can use official content.

Citibot does not publish a crawler page that explains visit schedules, page depth, or refresh rules. The directory records one user agent, CitibotSiteCrawler/1.0. Logs can show which government URLs it fetched, but the available operator material does not support assumptions about when it will return.

There is no Web Bot Auth signature URL or verified IP status attached to the record. CitibotSiteCrawler is useful for classifying a request, but any client can copy that header. Confirm unexpected traffic with Citibot before giving it access that would otherwise be denied.

These crawls can affect the information available in a participating agency's Citibot service. They are not documented as general web indexing or foundation-model training. A CitibotSiteCrawler visit therefore says nothing by itself about rankings in conventional search or inclusion in a public AI answer engine.