Known-Answer Evaluation: Why Generated Ground Truth Beats Labeled Ground Truth

Two ways benchmarks get their ground truth

Hand-labeling, done carefully, looks like this: an LLM seeds candidate questions from source material, and human annotators then filter, rewrite, and decompose those candidates into evidence statements before anything ships as an answer key. LongMemEval is a well-documented example of this process done well — 500 questions spanning five memory abilities, seeded and then manually reviewed and rewritten by annotators, with the resulting ground truth genuinely human-authored rather than machine-asserted.

Generation works differently in kind, not just in effort. A deterministic generator computes a scenario — in DecisionSynth Bench's case, a decision episode: trigger, options, recommendation, resolution, outcome — and writes the QA answer key as typed fields in the identical pass that creates the episode. There's no later moment where a person (or a model) reads the finished scenario and decides what the right answer is; the right answer is whatever the deterministic process computed, by definition, because the process itself is the source of truth rather than an interpreter of one.

The case for generated ground truth

Generated ground truth removes an entire category of error at the root: there's no separate labeling step to disagree with, misinterpret, or get wrong, because the label isn't an independent judgment about a scenario — it's the same computation that produced the scenario in the first place. That has two concrete benefits. Labeling has effectively zero marginal cost as the corpus grows, since the generator produces the answer key for free with every new episode it creates, where hand-labeling requires new annotator time for every new example. And there's no interpretive disagreement risk to manage — two annotators can read the same case note and reasonably disagree about what the correct answer is; a deterministic generator can't disagree with itself, because there's only one process, running once, producing one answer.

This isn't a claim that hand-labeling is careless or that LongMemEval's approach is inferior — a careful hand-labeling process, human review after LLM seeding, is a real and defensible way to manage the same underlying problem for domains that can't be generated. The comparison is about what each approach costs and where each one is actually available, not about rigor.

Where labeled ground truth remains necessary

Generated ground truth has a real constraint, and it's the boundary that keeps this argument honest: a generator can only emit ground truth for scenarios it can construct deterministically in the first place. That rules out an entire category of valuable question — genuinely ambiguous or subjective cases designed on purpose. An abstention question ('does the system correctly say it doesn't know') needs a human to design a scenario with no clean answer intentionally; a deterministic process, by construction, produces a scenario with a clean, computable answer, which means it structurally cannot manufacture the kind of genuine ambiguity an abstention test requires. DecisionSynth Bench doesn't have an abstention category for exactly this reason — it's a disclosed gap, not an oversight, and it's the clearest illustration of where hand-labeling remains the only tool that works.

The imitation-defense angle

Generated ground truth's near-zero marginal cost sounds like it should be a liability once you consider gaming: if answer keys are cheap to produce, are they also cheap to fake or imitate? The structural defense is a private held-out set — episodes generated by the identical process, over content (households, in DecisionSynth Bench's case) asserted disjoint from anything public. An imitator can, in principle, generate plausible-looking scenarios by studying the public examples, since the generation logic itself isn't secret. What it can't do is produce verified-correct answers against households it has never seen, because a plausible-looking answer isn't the same thing as a computed one — the private set is where 'known-answer by construction' actually gets tested against a system rather than just claimed.

A practical decision rule for benchmark builders

The choice comes down to one honest question about the domain itself: can the scenarios be constructed by a deterministic process end to end, with rules that are codifiable and outcomes that are computable from inputs? If yes — a financial decision governed by codified regulatory rules and a household's own numbers, for instance — generation removes labeling as an error source almost for free, and the upfront cost shifts to building and validating the generator itself rather than to per-example annotation. If the domain requires genuine human judgment, or the questions worth asking are specifically the ambiguous ones a computable process can't produce on purpose, invest in careful hand-labeling instead — seeded, reviewed, and decomposed by people, the way a rigorous hand-labeled benchmark does it. Neither answer is the 'more serious' one; they're the right tool for structurally different domains.

Key takeaways

• Hand-labeling and generation solve the ground-truth problem differently in kind — labeling is a person's interpretation of a finished scenario, generation is the same process that creates the scenario also emitting the answer.
• Generated ground truth removes labeling as a separate error source and drives marginal labeling cost toward zero as the corpus grows, but only for domains a deterministic process can actually construct.
• Careful hand-labeling (LLM-seeded, then human-reviewed and decomposed) remains legitimate and is the only option for genuinely ambiguous or subjective question types a computable process can't manufacture on purpose — abstention questions being the clearest example.
• A private held-out set is the structural defense against generated ground truth's low cost being exploited by imitation — plausible-looking generated content is not the same as verified-correct answers against genuinely unseen scenarios.
• The right approach follows from whether the domain itself is deterministically simulable, not from which method sounds more rigorous in the abstract.

FAQ