What is Twilio Knowledge?
Twilio Knowledge fetches website content when a Twilio customer adds a web source to a knowledge base. In Twilio's Enterprise Knowledge guide, sources sit inside a knowledge base whose ID is then connected to an AI tool.
A customer supplies the starting URL and a crawl depth. The current API supports depth values from 1 through 10 and recrawl periods of weekly, biweekly, monthly, or never. Twilio also accepts raw text and file uploads, so website crawling is one ingestion option rather than a requirement.
After ingestion, Twilio tools retrieve information from the knowledge base as context. The documentation names Twilio Agent Connect, Conversation Intelligence, language operators, and the Assistant API as consumers. This is a retrieval workflow for configured Twilio applications, not a public web index.
Twilio does not describe these website sources as foundation-model training data. Allowing the crawler helps the Twilio customer who configured that source keep its assistant or analysis workflow current. It does not make the page generally discoverable in AI search and does not promise a citation outside that customer's application.
The crawler needs a publicly reachable URL. Twilio documents ingestion failures when robots.txt or a firewall blocks the target, a server returns 403 Forbidden, or rate limiting produces HTTP 429. Customers can reduce crawl depth, lengthen the recrawl period, or upload files instead when direct crawling is unsuitable.
This entry records robots.txt compliance but has no stable user-agent token, full user-agent example, verified IP range, or HTTP signature directory. It is therefore marked unverifiable. Do not copy the old placeholder TwilioKnowledge into a bot-specific rule or allowlist; the supplied facts do not establish that token.
