Retrieval Evaluation

Retrieval evaluation is the measurement of how well a search or AI system finds the right evidence for a query: whether the relevant documents and passages were retrieved, how highly they were ranked, and whether irrelevant material crowded them out. It scores the step before the answer, which is why it is kept separate from LLM evaluation of the final response. A fluent answer built on the wrong sources is a retrieval failure, not a writing failure.

In information retrieval research the core metrics are well established. Recall@k is the share of all relevant items that appear in the top k results. Precision@k is the share of the top k results that are relevant. Mean reciprocal rank rewards placing the first relevant result high. nDCG@10, normalized discounted cumulative gain over the top ten, credits relevant results more the higher they rank and is the standard headline number in most current benchmarks; Voyage AI, for example, reported its September 2026 reranker results as nDCG@10 across 95 datasets.

Shared benchmarks make systems comparable. BEIR (Thakur et al., 2021) combined 18 public datasets to test how retrieval models generalize to unseen domains, and found BM25 a strong zero-shot baseline, with reranking and late-interaction models performing best on average at higher computational cost (Thakur et al.). The TREC Retrieval-Augmented Generation track, run in 2024 and 2025 on the MS MARCO v2.1 segment collection, separates three tasks: retrieval alone, generation from fixed retrieved segments, and end-to-end RAG with attributions (TREC RAG). Answers are scored with 'nuggets', the atomic facts a good answer must contain, which are now largely extracted and matched by language models and checked against human judgments.

AI search companies evaluate their own retrieval the same way. When Perplexity launched its Search API in September 2025, it published an open-source framework, search_evals, that scores search APIs by how well an AI agent answers SimpleQA, FRAMES, BrowseComp and Humanity's Last Exam questions using each API's results.

For GEO, retrieval evaluation is applied from the outside. You cannot inspect an engine's index, but you can observe what it retrieved for your prompts and compare that with what it should have found. Promptwatch separates the sources of each tracked response into inline citations, footnotes and search results, the last being pages the model browsed but did not cite, which lets teams see whether a missing citation was a retrieval problem (your page never appeared) or a selection problem (it appeared and lost). Useful checks include:

• For prompts where your page is the best answer, how often it appears anywhere in the retrieved set (owned recall).
• Where your page ranks among retrieved and cited sources, tracked over time.
• Whether the engine retrieves your current page or an outdated copy, archive or third-party summary.
• How many distinct domains the engine draws on for a topic, its citation diversity.

These checks turn 'we are not visible' into a diagnosis. Low owned recall points to crawling, indexing or retrieval coverage issues; high recall with low selection points to weaker passages or stronger competitor sources.