Is Devin a search crawler?

No public search-indexing role is documented. Its requests are associated with active engineering sessions rather than continuous discovery for a search engine.

Does a Devin visit affect AI training?

There is no documented training-data role for this bot. A request shows that a Devin session accessed the page, not that the page was added to a model-training corpus.

Does Devin follow robots.txt?

This catalog does not establish whether Devin follows robots.txt; Cloudflare's bot directory records the engineering agent as noncompliant. Use the `Devin` rule to express a preference, but enforce required restrictions outside robots.txt.

Can I verify Devin from its user agent?

No. The recorded header contains `Devin/1.0`, but there is no Web Bot Auth directory or verified IP identity listed. Treat the string as a log classification signal only.

Devin: AI Crawler Guide & robots.txt

What is Devin?

Devin is an AI software engineer that can write, run, and test code for an engineering team. Users assign it work such as implementing a ticket, reproducing a bug, reviewing a codebase, or building an integration. The Devin web identity belongs to browsing performed as part of that engineering work, not to a standalone search engine.

When Devin requests a page, the URL is usually connected to a current task. It may be reading API documentation needed for an implementation or exercising a web flow that it has been asked to test. That request pattern is narrower and more situational than a crawler that continuously discovers pages for a shared index.

Cloudflare's bot directory records this browser-style identity as Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36; Devin/1.0; +https://devin.ai. The stable matching token is Devin. The official product overview is available at https://docs.devin.ai/get-started/devin-intro.

Allowing Devin does not make a site eligible for a public AI search result, and blocking it does not opt the site out of model training. The documented impact is on the engineering session that needed the page. If Devin cannot reach required documentation or a test application, that assigned task may be incomplete.

Devin sessions include an interactive browser alongside the shell and code editor. The browser can read documentation, test web applications, capture evidence, and move through multi-step interfaces. A user can watch the session and take over when a login, CAPTCHA, or other manual step needs attention.

Robots.txt behavior is unknown, and Cloudflare's bot directory lists the bot as not following robots.txt. Devin has no Web Bot Auth directory or verified IP identity in this bot, so a matching header cannot prove who sent the request. Sensitive documentation and test systems should continue to require authentication and task-appropriate authorization.

Relevant for AI search

Is Devin relevant for AI search?

Yes. Devin collects pages for an AI product, so what it can crawl influences how AI systems describe your brand.

Devin crawls and indexes pages so the AI search or assistant behind it can retrieve them at answer time. A page it has never fetched cannot be quoted, summarized, or linked in that product's answers, so most sites keep it allowed to stay citable. Blocking it removes your pages from that AI surface and hands those citations to competitors.

How to handle Devin

Public technical documentation can remain available if developers are expected to use Devin while working with your product. For staging applications, account pages, and private docs, rely on the same login and permission checks used for a human engineer. The bot name is not an authorization credential.

State a no-crawl preference with:

User-agent: Devin
Disallow: /

Devin AI has not published a robots.txt commitment for this identity, so check logs after adding the rule. If automated browser access must be denied, enforce that policy at the application or edge and account for the fact that the Devin header can be spoofed.

Examples

Devin reads a library's API reference while implementing an integration requested by an engineering team.
A developer asks Devin to test a staging checkout flow, and the application grants access through its normal test account rather than a bot allowlist.
An administrator sees the `Devin/1.0` token on an unexpected private route and investigates because the header cannot authenticate the caller.

Frequently asked questions about Devin

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Devin uses an interactive browser while completing software-engineering tasks. It may read documentation, test a web application, collect a screenshot, or navigate a flow needed for the assigned work.

Related bots

Amazon Kendra

Amazon Kendra is a managed information retrieval and intelligent search service that uses natural language processing and advanced deep learning model.

AI Assistant