Promptwatch Logo

Terracotta

The Terracotta bot scrapes websites for use in generating indices for serving searches using Ceramic's search product.
CeramicTerracotta
Search Engine Crawler

What is Terracotta?

Terracotta appears in the catalog under two user-agent values: Terracotta and Terracotta-News. The record does not explain how their scopes differ. Keeping the full value in log reports lets a publisher review the news-labeled traffic separately without guessing what the label means.

Both identities belong to Ceramic's crawl for its search product. Terracotta scrapes websites and supplies pages to an index that Ceramic uses to serve searches.

The robots.txt picture is unsettled. Compliance is not confirmed in the bot metadata, and Cloudflare's bot directory marks it as false. There is no verified IP range or request-signature directory either, so neither header proves who sent a request.

A Terracotta hit confirms that the crawler reached a URL for index generation. It does not show that Ceramic kept the page or served it to a search user. The record assigns no AI-answer or model-training purpose to this bot, so Ceramic's search index should not be presented as a generative AI corpus.

Relevant for AI search

Is Terracotta relevant for AI search?

Yes. Terracotta collects pages for an AI product, so what it can crawl influences how AI systems describe your brand.

Terracotta feeds a search index that also powers that engine's AI answer features and overviews. One crawl can serve a classic results page and a generated summary, so blocking it costs you both the rankings you would expect and a growing share of AI answers built on the same index.

How to handle Terracotta

Leave Terracotta able to reach public pages if inclusion in Ceramic's search product is useful. To state an opt-out for both recorded identities, use:

User-agent: Terracotta
User-agent: Terracotta-News
Disallow: /

Cloudflare's bot directory records Terracotta as not following robots.txt, so confirm the result in access logs. Apply an edge or application rule when exclusion is mandatory. Since the catalog provides no authenticated source identity, do not use either user-agent value as an access credential.

Examples

  • A publisher separates `Terracotta-News` requests from ordinary `Terracotta` traffic to see which article URLs each identity fetches.
  • A site owner who does not want pages collected for Ceramic search publishes rules for both recorded user agents and verifies the outcome at the edge.

Frequently asked questions about Terracotta

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Ceramic uses Terracotta to scrape websites and generate indices that serve searches in Ceramic's search product.

Related bots

AdIdxBot

An AdIdxBot visit usually starts with a Bing Ads workflow rather than an organic search crawl.

Search Engine Crawler

Adsense

The AdSense crawler visits participating sites in order to provide them with relevant ads.

Search Engine Crawler

adsnaver

adsnaver is tied to paid search advertising, not Naver's general web index.

Search Engine Crawler

Algolia

The Algolia Crawler extracts content from your site and makes it searchable.

Search Engine Crawler

Amazon AdBot

Amazon AdBot scans pages that request ads from Amazon's advertising systems.

Search Engine Crawler

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.

Promptwatch Dashboard