Known-Answer Benchmark
A known-answer benchmark is one where the ground-truth answer is emitted by the same deterministic process that creates the test scenario — a generator, not a human annotator or a model — so the answer key is correct by construction rather than by interpretation after the fact.
Most benchmarks build ground truth the other way around: a scenario or document exists first, and a human (sometimes assisted by an LLM seed) reads it and decides what the correct answer is. LongMemEval is a well-documented example of that approach done carefully — questions are seeded by an LLM, then human annotators manually filter, rewrite, and decompose the answers into evidence statements (Wu et al., 2025, §3.2). That process produces real, defensible ground truth, but it's still an interpretive step: a person is deciding what counts as correct, and that decision carries the same disagreement risk any human judgment call carries.
A known-answer benchmark removes that step entirely. DecisionSynth Bench's generator computes a decision episode — trigger, options, recommendation, resolution, outcome — and writes the QA answer key as typed fields in the same pass, with no LLM anywhere in the generation path. There's no separate moment where someone reads the finished episode and decides what the right answer is; the right answer is whatever the deterministic process computed, by definition.
The benefit is twofold: labeling has effectively zero marginal cost as the corpus grows (the generator produces the answer key for free with every new episode), and there's no interpretive labeling step to disagree with, since the answer key is definitionally what the generator computed rather than an independent judgment about it. The constraint is real too — a generator can only emit ground truth for scenarios it can construct deterministically, which rules out genuinely ambiguous or subjective questions (an abstention test, for instance, needs a human to design a question with no clean answer on purpose).
"Known-answer by construction" describes the labeling step only — it doesn't mean the generator can't have bugs. DecisionSynth Bench's episodes still pass a schema and cross-field-invariant gate before publication, with golden fixtures pinned in CI; a gate failure is treated as a generator bug to fix, not something to retry around. What known-answer-by-construction guarantees is that there's no additional, separate error source layered on top of whatever the generator itself gets right.
Common pitfalls
- Assuming 'known-answer by construction' means the benchmark is infallible — it means the labeling step is error-free, not the generation step; a generator bug is still possible and is caught by validation, not by the answer-key mechanism.
- Assuming this approach generalizes to any domain — it only works where the test domain can be deterministically simulated in the first place; open-ended or genuinely subjective questions still need human-authored ground truth.