DecisionSynth Bench's held-out set has 1,367 QA tasks. 591 of them are direct recall. 182 are temporal ordering. 9 are precedent search — the rarest of the five task types, because the trigger condition it depends on (a policy override for a specific, matchable reason, with prior similar cases to search against) is itself the rarest pattern in the underlying decision corpus. Every scoreboard row reports a precedent-search exact-match number computed from those 9 tasks, sitting in the same table, formatted the same way, as columns computed from hundreds.
That's an uncomfortable number to publish next to columns fifty or a hundred times larger, formatted with the same decimal precision as if it carried the same statistical weight. The alternative — dropping the column, or quietly excluding it from headline reporting — would look tidier. It would also hide a real, if narrow, signal, and would let anyone reading the scoreboard assume coverage that doesn't exist.
Why the column is this small in the first place
Precedent search asks: which other episodes overrode policy for the same reason, and what happened to them. Answering that requires the corpus to contain multiple episodes sharing a specific override reason — which only happens when a household hits an override-eligible trigger more than once, or when several households in the same archetype cohort happen to override for matching reasons within the sampling frame. DecisionSynth Bench's coverage rule targets at least two episodes per archetype per applicable decision type across the full corpus, which is enough to guarantee the task type exists but not enough to guarantee it exists in volume — especially in a held-out slice roughly the size of the dev set. The dev set's own precedent-search column, over a separately sampled set of episodes, comes in at 11 tasks — close to, but not identical to, the held-out count, which is itself evidence that "small" is a structural property of this task type's rarity, not an artifact of one particular sampling run.
| Task type | Held-out task count | What triggers it | |
|---|---|---|---|
| Direct recall | 591 | Every episode has an outcome | |
| Rule attribution | 591 | Every episode cites governing figures | |
| Temporal ordering | 182 | Multiple episodes for the same household within a window | |
| Rationale lookup | ~112 (dev-set scale) | Only overridden episodes carry a reason | |
| Precedent search | 9 | Multiple episodes sharing the same override reason |
What 9 tasks can tell you
A 9-task column is not statistically meaningless. The naive-lexical baseline scores 1.000 exact-match on precedent search in both the dev and held-out runs — a system that stores everything verbatim and searches by keyword overlap correctly identifies every precedent in a 9-task sample, which is a legitimate, if narrow, confirmation that the ceiling reference behaves as expected even on this task type. Letta's archival configurations score 0.667 on the same held-out column — a real, specific number that says something about how the system performs when it has to aggregate across records rather than answer from one.
What a 9-task column earns you is a directional read: is a system's performance on this task type roughly consistent with its performance elsewhere, or does it look like an outlier worth a closer look. It does not earn you a claim like "System A beats System B at precedent search by 15 percentage points" stated with the same confidence a 591-task column would support — the margin of a 9-task binary-outcome column is wide enough that a difference of one or two correct answers swings the reported percentage by more than 10 points.
The alternative was worse
Three less-honest ways to handle a 9-task column were available, and each was rejected for a specific reason:
- Drop the column entirely. This would understate what the benchmark measures — precedent search is a real, decision-relevant capability (which past cases resemble this one, and what happened to them), and omitting it entirely because it's inconvenient to report responsibly is a worse failure than reporting it with a caveat.
- Fold it into the overall average without flagging it. This would let a 9-task result quietly move a system's headline number by roughly the same weight as any other task type, overstating its statistical contribution relative to its actual sample size.
- Report it with the same confidence language as the larger columns. This is the most tempting failure mode, because it requires no extra words — just silence about sample size — and it's the one most likely to mislead a reader who reasonably assumes every column in a table carries comparable weight unless told otherwise.
precedent_search has only 9 held-out tasks (rarest trigger type in the coverage frame) — treat that column as directional, not significant.
Publishing the column with that explicit caveat, directly in the scoreboard rather than buried in a methodology appendix, is the version that respects the reader's ability to weight the number correctly instead of doing the weighting silently on their behalf.
The general practice this argues for
Any benchmark with multiple task types is going to have an "n=9" column somewhere, because real-world task frequency is never uniform — some questions are simply rarer than others in whatever corpus or production log the benchmark draws from. The choice isn't whether to have a small column; it's whether to report it honestly. That means: publish the actual count next to the score, say explicitly when a column should be read as directional rather than significant, and resist the urge to either hide the column or dress it in the same statistical clothing as the columns that earned it.
Key takeaways
- DecisionSynth Bench's held-out precedent-search column rests on 9 tasks — the rarest of five task types, because its trigger condition (multiple episodes sharing an override reason) is itself the rarest pattern in the corpus.
- A 9-task column supports directional reads (is this system roughly consistent with its other scores, or an outlier) but not fine-grained ranking claims — each task is worth roughly 11 percentage points, so single-task swings shouldn't be read as meaningful differences.
- The alternatives — dropping the column, folding it into an unflagged average, or reporting it with unwarranted confidence — were each judged worse than publishing it with an explicit small-n caveat.
- Any multi-task-type benchmark will have a rare column somewhere; the discipline is reporting the actual count and treating it accordingly, not avoiding having one.