What data does MJ12bot collect?

Majestic says it currently builds a graph of links between websites for its backlink search index. Its bot page says it does not currently cache web content or personal data.

Does MJ12bot respect robots.txt?

Majestic says yes and documents supported directives, including `Crawl-delay`. The Cloudflare record conflicts with that claim, so confirm that disallowed paths stop receiving requests.

Why does MJ12bot keep requesting a 404 or redirect?

Its documentation says linked URLs are revisited so temporary errors or redirects do not permanently alter a site's link profile. Persistent links elsewhere can keep an old URL in the crawl plan.

Will rel=nofollow stop MJ12bot from fetching a link?

No. Majestic explains that `nofollow` is not a crawl prohibition. Disallow the destination in `robots.txt`, require authentication, or enforce a server rule if it must not be fetched.

Can MJ12bot be verified by IP address?

Not with a compact fixed range because the crawler is distributed. Majestic offers a site-specific `CRAWLER-IDENT` value on request for operators who need stronger attribution.

Does MJ12bot improve AI search visibility?

There is no direct documented effect. It supplies Majestic's link index and SEO tools, not a named AI answer or model-training system.

MJ12bot: AI Crawler Guide & robots.txt

What is MJ12bot?

No definition available.

Relevant for AI search

Is MJ12bot relevant for AI search?

Yes. MJ12bot collects pages for an AI product, so what it can crawl influences how AI systems describe your brand.

MJ12bot feeds a search index that also powers that engine's AI answer features and overviews. One crawl can serve a classic results page and a generated summary, so blocking it costs you both the rankings you would expect and a growing share of AI answers built on the same index.

How to handle MJ12bot

Allow MJ12bot if you are comfortable contributing link relationships to Majestic's index. If its request volume is the concern, Majestic documents Crawl-delay values up to 20 seconds; copy that directive into the MJ12bot section because a bot-specific section takes precedence over the wildcard section.

To stop the crawler, use:

User-agent: MJ12bot
Disallow: /

Majestic's documentation and the bot metadata say MJ12bot honors robots.txt, while Cloudflare's matching record marks compliance false. Verify the outcome in logs. Majestic also notes that the crawler must be able to retrieve /robots.txt; if that request is blocked, it may assume crawling is allowed. For a hard denial, enforce the policy at the server or edge.

Examples

An SEO analyst leaves MJ12bot enabled so new links to a product guide can eventually appear in Majestic's backlink tools.
A small documentation server adds `Crawl-delay: 5` to its MJ12bot record after distributed requests create short bandwidth spikes.
An administrator sees repeated requests for an old redirected URL and checks inbound links before treating the traffic as a broken crawler loop.