Term

Evidence Retrieval (Precision/Recall@k)

Published Jul 18, 2026

Definition

Precision@k and recall@k are the retrieval half of a two-axis memory-evaluation score: precision@k is the fraction of the k items a system retrieved that were actually relevant, and recall@k is the fraction of all relevant items that the system's retrieval surfaced — scored separately from whether the system's final answer was correct.

A memory system can fail in two independent ways: it can retrieve the wrong evidence, or it can retrieve the right evidence and still answer incorrectly. Collapsing both into one pass/fail number hides which failure is happening, which is why retrieval gets scored on its own axis, before answer correctness is scored separately.

The two metrics answer different questions at a fixed retrieval budget k. Precision@k asks: of the k items returned, how many were actually relevant? Recall@k asks: of all the relevant items that existed, how many did the system find? A system can score high on one and low on the other — returning a lot of noise alongside the right answer (low precision, maybe-fine recall), or missing most of what mattered while what it did return was on-target (fine precision, low recall).

One structural detail matters when reading a precision number: for any task with a single relevant piece of evidence, retrieving k=5 items caps precision at 0.2 by construction (1 relevant item out of 5 returned) — a bound built into the arithmetic, not a sign of a weak system. DecisionSynth Bench's verbatim-storage baseline scores P@5 0.292 and R@5 0.972 on the dev set (held-out: P@5 0.302, R@5 0.994); read on its own, 0.292 looks unimpressive next to a 0.972 recall figure on the same row, until the single-evidence-task ceiling is accounted for, at which point it's close to the maximum achievable once multi-evidence task types are folded into the average.

Formula

Precision@k / Recall@k

P@k = relevant_retrieved / k · R@k = relevant_retrieved / total_relevant

relevant_retrieved: = count of returned items that are actually relevant
k: = the fixed number of items retrieved
total_relevant: = count of items that were relevant, whether retrieved or not

Example

1 relevant item, k=5, retrieved correctly → P@5 = 1/5 = 0.2 (the structural ceiling for a single-evidence task), R@5 = 1/1 = 1.0

Precision and recall are scored independently of whether the final typed answer was correct.

Why this matters for synthetic data

Comparing precision or recall across systems is only fair at a matched k — and even then, systems with different native retrieval-unit granularity (a whole passage vs. one extracted fact vs. one graph edge) aren't directly comparable at the same raw k, since a fixed budget carries a very different amount of source material depending on the unit size.

Common pitfalls

Reading a precision number as a weakness without checking how many relevant items the underlying task actually had — a single-evidence task structurally caps precision@5 at 0.2.
Comparing retrieval scores across systems with different native granularities (passages, facts, graph edges) at the same raw k without accounting for what k represents in each.

Frequently asked questions

Which matters more, precision or recall?+

It depends on the cost of each error type for the use case. A system that returns noisy evidence alongside the right answer (low precision) wastes a reviewer's time; a system that misses relevant evidence entirely (low recall) can produce a confidently wrong answer. Compliance-adjacent uses tend to weight recall heavily — missing the governing citation is worse than including an extra one.

Why score retrieval separately from answer correctness at all?+

Because the two failures point at different fixes. Poor retrieval with an otherwise-correct answer format suggests the memory pipeline isn't surfacing the right content; good retrieval with a wrong answer suggests a reasoning or extraction problem downstream of retrieval. A single blended score can't tell you which one to work on.