Promptwatch Logo

Marfeel Audits Crawler

Marfeel's audit crawlers that periodically re-crawl traffic-receiving URLs to detect structured data, meta tags, and HTML issues.
Marfeel
MonitoringAnalyticsSEO

What is Marfeel Audits Crawler?

Marfeel's Audits crawler checks client pages for structured data, meta tags, and other problems in the HTML. It periodically revisits relevant URLs that receive traffic rather than exploring a domain to discover content on its own. Marfeel works from canonical URLs and their linked AMP versions.

Two crawler identities divide the work. mrfCompass-Booldog first requests each URL with a mobile user agent and makes a desktop request when the response varies by user agent. mrfCompass-Marshall follows the amphtml links found by Booldog and audits those AMP pages.

Marfeel says its bots limit concurrent requests, cache assets for reuse, and rate-limit repeat crawling. The bot record marks robots.txt support as true, and Marfeel states that all of its crawlers follow the site's rules. Marfeel also documents forward-confirmed reverse DNS under crawler.marfeel.com for checking claimed traffic.

These requests feed Marfeel's site technology audits for its publishing customers. They are not documented as collection for a general AI search engine or foundation-model training. Fixing an issue found by the audit may help the site's ordinary technical SEO, but the crawl itself does not create AI visibility or citations.

Indirectly relevant

Is Marfeel Audits Crawler relevant for AI search?

Indirectly. Marfeel Audits Crawler has no AI product of its own, but its output can end up in the systems that AI answers draw on.

Marfeel Audits Crawler is an SEO auditing crawler. It has no direct effect on search rankings or AI answers, but the reports it feeds are used to fix crawlability and content issues that do affect how search and AI systems see your site.

How to handle Marfeel Audits Crawler

Publishers using Marfeel should allow both audit identities on the canonical and AMP pages they want checked. If a firewall intervenes, verify the source with a reverse DNS lookup to crawler.marfeel.com followed by a forward lookup to the original address before creating an exception.

To stop only the Audits crawler, use the two user-agent products from Marfeel's documentation:

User-agent: mrfCompass-Booldog
Disallow: /

User-agent: mrfCompass-Marshall
Disallow: /

This does not cover Marfeel's Editorial, Flowcards, Social, Amplify, or Preview crawlers, which use other tokens. Keep restricted content behind authentication unless access by Marfeel is an intentional part of the publishing setup.

Examples

  • A publisher sees Booldog fetch a traffic-receiving article on mobile, followed by a desktop request because the response includes `Vary: User-Agent`.
  • Marshall requests an AMP URL after Booldog finds the page's `amphtml` link, letting Marfeel audit the alternate version.
  • A WAF blocks Marfeel audits until the operations team verifies `crawler.marfeel.com` with forward and reverse DNS checks.

Frequently asked questions about Marfeel Audits Crawler

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Marfeel operates the audit crawlers for publishers using its analytics and site technology products.

Related bots

Marfeel Flowcards Crawler

Marfeel's crawler that fetches content for Flowcards that load directly from specific URLs.

Preview

Marfeel Preview Crawler

Marfeel's previewer crawler used to render preview experiences for both mobile and desktop views.

Preview

Marfeel Social Crawler

Marfeel's crawler used for social experiences (Facebook, X/Twitter, Telegram, Reddit, LinkedIn).

Preview

AhrefsBot

Powers the database for both Ahrefs, a marketing intelligence platform, and Yep, an independent, privacy-focused search engine.

SEO

AhrefsSiteAudit

Powers Ahrefs’ Site Audit tool. Ahrefs users can use Site Audit to analyze websites and find both technical SEO and on-page SEO issues.

SEO

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.

Promptwatch Dashboard