Held-Out Evaluation
Held-out evaluation is testing a system against a data split that was deliberately withheld during development — a private set of scenarios and answer keys the system under test could not have seen, trained against, or pattern-matched in advance.
Public benchmark data has a shelf life. Once a corpus and its answer keys are published, they can end up inside a model's training data, or a team can iterate against the visible answers until their system's score stops reflecting general capability and starts reflecting familiarity with that specific corpus. Held-out evaluation is the standard defense: keep a second split with the same task shape but different underlying content, never publish its answers, and run the scoreboard on that split instead of the public one.
DecisionSynth Bench implements this as two corpora. The public dev set — 591 episodes, 1,487 QA tasks, CC BY 4.0 with full answer keys — exists for free iteration; any team can build and tune an adapter against it. The held-out set — 543 episodes, 1,367 QA tasks — sits over households asserted disjoint from the dev set at build time, and its episodes and answer keys are never published. The public scoreboard runs on the held-out set specifically, not the dev set, precisely so a system's published score can't come from having seen the questions before.
Held-out evaluation earns its keep by catching things dev-only testing misses. On DecisionSynth Bench, the ranking order (verbatim storage ahead of extraction ahead of graph pipelines) held from dev to held-out — evidence the overall picture generalizes. But individual numbers moved a lot underneath that stable ranking: one extraction pipeline's rationale-lookup score collapsed from 0.402 on the dev set to 0.038 on held-out even as its overall score rose. A dev-only evaluation would never have surfaced that swing, because there would have been no second corpus to check it against.
A benchmark with only a public split — answers included — tells you how a system performs on data it could have been tuned against, not how it performs on genuinely new content. A held-out or private-test design matters most once a score is going to be published or used to compare vendors, rather than for early internal iteration.
Common pitfalls
- Assuming a strong dev-set score predicts a proportionally strong held-out score — per-task-type numbers can move substantially between the two even when the overall ranking holds.
- Treating "held-out" as a synonym for "harder" — it's not about difficulty, it's about whether the system being scored could have seen the specific content beforehand.