How does the Marginalia crawler identify itself?

Its recorded product token and user-agent value are `search.marginalia.nu`. Marginalia's webmaster page also publishes source network details that should be checked there before use.

Does Marginalia Search respect robots.txt?

Marginalia says yes, including specific and wildcard records, although the directory sources conflict. Changes can take time to reach an existing crawl plan, so verify over a reasonable interval.

Does Marginalia use sitemaps and feeds?

Its crawler documentation says it reads sitemap directives, checks `/sitemap.xml`, and can parse RSS or Atom feeds linked from the root page.

Can I slow the Marginalia crawler down?

Marginalia says it honors `Crawl-delay` and HTTP 429 responses. Its normal timing also adapts within limits to how quickly the server responds.

Is Marginalia Search an AI crawler?

No. Marginalia's current search page explicitly says its custom index and crawler use simple technology with no AI.

Marginalia Search AI Crawler Guide

What is Marginalia Search?

Marginalia Search looks for parts of the web that large commercial indexes often bury, especially personal sites, older pages, and independent blogs. It is a noncommercial search engine with its own crawler and index, and its current site says it prioritizes noncommercial content.

Marginalia explicitly describes its search stack as simple technology with no AI. Allowing the crawler can make a site available in Marginalia's niche search results, but there is no documented model-training benefit or general AI answer exposure.

The crawler identifies itself with search.marginalia.nu. Marginalia has described a planned crawl workflow in which a domain and a list of URLs are prepared before fetching begins. It can also discover URLs from sitemap directives, a root /sitemap.xml, and RSS or Atom feeds linked from a site's root page.

Marginalia publishes crawler network details for webmasters and says it adjusts request timing to the server's response speed. Its crawler notes also say it honors Crawl-delay and reacts to HTTP 429 responses. These are better controls for a slow site than trying to guess at crawler behavior from a single burst in the logs.

Relevant for AI search

Is Marginalia Search relevant for AI search?

Yes. Marginalia Search collects pages for an AI product, so what it can crawl influences how AI systems describe your brand.

Marginalia Search feeds a search index that also powers that engine's AI answer features and overviews. One crawl can serve a classic results page and a generated summary, so blocking it costs you both the rankings you would expect and a growing share of AI answers built on the same index.

How to handle Marginalia Search

Personal sites and small publishers may choose to allow Marginalia because that is the material its search engine sets out to surface. If only part of a site is suitable, use path-specific directives and keep a sitemap or feed available for the public section.

To block all Marginalia crawling, add:

User-agent: search.marginalia.nu
Disallow: /

Marginalia's webmaster guidance says it follows both rules for search.marginalia.nu and general rules. It warns that changes may not take effect immediately. The directory record leaves compliance unconfirmed and Cloudflare's bot directory marks it false, so check later requests and use an edge rule if you need immediate enforcement.

Examples

The owner of a long-running personal blog keeps the crawler allowed and exposes an RSS feed so Marginalia can find posts that are not heavily linked elsewhere.
A small server returns HTTP 429 during a traffic spike and reviews whether Marginalia backs off before adding a stricter crawl delay.

Frequently asked questions about Marginalia Search

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

It focuses on noncommercial and less visible parts of the web, including personal sites, older websites, and blogs. Inclusion and ranking are still decisions made by its search system.