Live session Nov 5: Fireflies' AI Search Playbook
Promptwatch Logo

FirecrawlAgent

Firecrawl's crawler, used by customers of its scraping API to convert websites into LLM-ready data.
UnverifiableFirecrawlFirecrawlAgent
AI CrawlerAgent

What is FirecrawlAgent?

FirecrawlAgent is the token Firecrawl matches against when it applies robots.txt rules. Firecrawl sells an API that crawls and scrapes sites for developers building AI applications. Each crawl is started by a customer, so volume depends on who points the service at your domain.

According to the Firecrawl docs, it reads robots.txt before crawling and applies rules written for FirecrawlAgent as well as rules for *. Customers can only ignore robots.txt on an Enterprise plan.

The docs warn against assuming FirecrawlAgent is the User-Agent header on the fetch itself. Scrape requests may send FireCrawlAgent/1.0.0 instead, and Firecrawl does not use a fixed set of outbound IP addresses. That makes the request hard to verify from logs alone.

Indirectly relevant

Is FirecrawlAgent relevant for AI search?

Indirectly. Firecrawl turns pages into LLM-ready data for its customers, so the effect depends on what they build.

Firecrawl is a scraping and crawling API that developers use to feed web content to AI agents, RAG systems and datasets. It has no consumer AI product of its own, so allowing or blocking it does not change your standing in a specific answer engine. It does decide whether Firecrawl customers can read your pages.

How to handle FirecrawlAgent

Firecrawl recommends controlling it with a robots.txt directive rather than a WAF rule that matches a user-agent string. To block it:

User-agent: FirecrawlAgent
Disallow: /

To allow it while blocking other bots, give it its own group with Allow: /. Because Firecrawl has no fixed IP ranges and customers on Enterprise can ignore robots.txt, add rate limits at the edge if the traffic matters.

Examples

  • A developer points Firecrawl at a documentation site to build a RAG index, and the site sees a burst of sequential page fetches.
  • A site owner disallows `FirecrawlAgent` in robots.txt to keep its content out of third-party AI pipelines.

Frequently asked questions about FirecrawlAgent

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Yes by default. Firecrawl applies the `FirecrawlAgent` and `*` groups. Enterprise customers can enable an option that ignores robots.txt.

Related bots

Apify Website Content Crawler

Crawl websites and extract content to feed AI apps. Convert web data to Markdown or HTML, download files, and more.

AI Assistant

Browserbase

Runs headless browser automation on behalf of Browserbase customers for web scraping, form submission, and testing.

AI Crawler

Cloudflare Crawler

Cloudflare Crawler is the bot identity used by the /crawl endpoint in Cloudflare Browser Rendering.

AI Crawler

Diffbot

Diffbot crawls and structures web pages into a knowledge graph that is sold for AI training, retrieval, and data enrichment.

AI Crawler

ExaSearchBot

Exa's search crawler, which discovers and indexes public web pages for its search and retrieval API.

AI Search

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.

Promptwatch Dashboard