Promptwatch Logo

Jina Reader

Jina AI's reader service, which fetches one URL on request and returns the page as clean markdown for LLMs.
Jina AIJinaReader
AI AssistantUser InitiatedAgent

What is Jina Reader?

Jina Reader is the service behind r.jina.ai. It takes a single URL, loads the page, extracts the main content and returns it as markdown that a language model can read. The JinaReader bot page says it fetches a page only when a person or an application asks for that specific URL. It does not discover URLs, follow links or crawl sites.

Depending on the page and the request, Reader loads the URL in a headless browser or with a plain HTTP request. The requester decides what happens to the result, which in practice usually means feeding it to an LLM app, an agent or a RAG pipeline. Reader is open source, and the code lives at github.com/jina-ai/reader.

Identification works through Web Bot Auth rather than a fixed user-agent string. Signed requests carry Signature-Agent: agent="https://r.jina.ai" with Signature-Input and Signature headers, and the keys are published at https://r.jina.ai/.well-known/http-message-signatures-directory. Jina says requests without these headers do not claim to be JinaReader.

robots.txt handling depends on that signature. For signed requests, Reader fetches /robots.txt first and applies the JinaReader group, falling back to *. An authenticated account can switch the check off; anonymous callers cannot. Unsigned requests are only checked against robots.txt if the caller asks for it, which is why compliance is listed as partial here.

Relevant for AI search

Is Jina Reader relevant for AI search?

Yes. Jina Reader turns a page into markdown for the LLM apps and agents that call it, so it decides what those tools can read from your site.

Many AI apps and agents do not fetch pages themselves. They send the URL to Jina Reader and pass the markdown it returns to a language model. If Reader cannot load a page, or loads it with the content missing, the model answers without your text. Blocking JinaReader keeps signed requests out, but it also takes your pages out of every tool that reads the web through Reader.

How to handle Jina Reader

Leave Jina Reader allowed if you want LLM tools and agents that use it to read your pages. To keep signed Reader requests away from the whole site:

User-agent: JinaReader
Disallow: /

That rule covers signed requests from anonymous callers. Unsigned requests, and callers with an account who disable the check, are not bound by it. For a hard block, verify the Web Bot Auth signature at your CDN or WAF and reject the requests you do not want.

If Reader returns empty or broken markdown for your pages, check that the main content renders without a login wall, consent overlay or heavy client-side loading. Those are the same problems that stop other AI fetchers.

Examples

  • An AI agent sends a pricing page URL to `r.jina.ai`, gets markdown back, and quotes the plans in its answer.
  • A developer notices signed JinaReader requests in the logs and traces them to a RAG pipeline that ingests the company's docs.
  • A publisher adds a `JinaReader` group to robots.txt and confirms that signed requests stop while its search crawler rules stay unchanged.

Frequently asked questions about Jina Reader

Learn about AI visibility monitoring and how Promptwatch helps your brand succeed in AI search.

Be the brand AI recommends

Monitor your brand's visibility across ChatGPT, Claude, Perplexity, and Gemini. Get actionable insights and create content that gets cited by AI search engines.

Promptwatch Dashboard