Precedent Search
Precedent search is a memory-evaluation task that asks a system to find every other record sharing a specific characteristic with a current one — such as other decisions overridden for the same reason — and report the outcome of each, rather than retrieving a single matching record.
DecisionSynth Bench's precedent_search task type asks which episodes overrode policy for a given reason, and with what outcomes, with the answer key an episode-id set plus each episode's status. It's the task type that most directly tests aggregation: a system can retrieve the current episode perfectly and still fail precedent search if it can't search across the rest of the corpus for matching records, or if its extraction step never preserved the override-reason field as a comparable, searchable value in the first place.
It's also structurally the rarest task type in the benchmark. Answering it requires the corpus to contain multiple episodes sharing a specific override reason, which only happens when a household hits an override-eligible trigger more than once, or when several households in the same archetype cohort happen to override for matching reasons within the sampling frame. The held-out scoreboard's precedent-search column rests on 9 tasks — out of 1,367 total, the smallest of the five task types by a wide margin (the dev set's separately-sampled column comes in similarly small, at 11).
That small sample size doesn't make the column meaningless, but it does bound what it can support. The verbatim-storage baseline scores a clean 1.000 exact-match on precedent search in both the dev and held-out runs, confirming the ceiling reference behaves as expected even on this task type; an archival system's held-out score of 0.667 is a real, specific data point about aggregation performance. What a 9-task column can't support is a fine-grained ranking claim — at that sample size, one or two flipped answers swing the reported percentage by more than 10 points, so the honest reading is directional (roughly consistent with a system's other scores, or a notable outlier), not a precise ranking.
A benchmark's rarest task types deserve the same disclosure discipline as its largest: publish the actual task count next to the score, and read a small column as directional rather than as support for a precise ranking claim.
Common pitfalls
- Treating a small-sample column (single digits to low tens of tasks) with the same statistical confidence as a column built from hundreds of tasks.
- Assuming a system that fails precedent search overall has weak memory — it may simply not aggregate across records well, a distinct capability from single-record retrieval.