Answer Exact-Match (EM)
Answer exact-match (EM) is a scoring method that compares a system's typed answer to a ground-truth answer key and credits the task only when the values match exactly — as opposed to an LLM-judged score, which asks a model to rate how correct an answer seems.
A typed field usually has exactly one correct value: a decision's outcome either was 'approved' or it wasn't; the cited rule either was retirement.401k.elective_deferral.under_50 or it wasn't. Exact match reflects that directly — it checks the system's answer against the ground-truth value for that field and records a binary correct/incorrect, with the comparison shape matched to the question: an unordered set match for a 'which of these episodes' question, an ordered match for a sequence question, and a plain field match everywhere else.
The alternative — asking an LLM to judge whether an answer 'looks about right' — trades a hard comparison for a soft one, and that softness is a real cost in a domain where the underlying question has one correct answer. An LLM judge can disagree with itself between runs, and its notion of 'reasonable' isn't the same thing as 'matches the record.' For a field a compliance reviewer would check against a specific document, exact match is the standard that reflects what's actually being asked; DecisionSynth Bench's baseline scores 0.950 overall EM on the dev set and 0.980 on the held-out set, numbers that mean something precise because the comparison behind them is precise.
Exact match becomes the wrong tool only when the field itself has legitimate paraphrase room — genuine free text where several different phrasings would all be correct. That's a signal to redesign the field as a typed value upstream, not a reason to loosen the scoring downstream; softening the metric to accommodate an underspecified field just hides the design problem instead of fixing it.
| Exact match (EM) | LLM-judged scoring | |
|---|---|---|
| What it checks | Does the typed value match the ground-truth key exactly | Does a judge model rate the answer as roughly correct |
| Consistency | Deterministic — same inputs, same result | Can vary between runs of the same judge |
| Where it fits | Fields with exactly one correct value | Open-ended text with no single correct phrasing |
Teams designing their own evaluation schema should ask, for every field: is there exactly one correct value here? If yes, type the field and score it with exact match. If the honest answer is 'several phrasings would be correct,' that's a sign the field needs redesigning — not a reason to reach for LLM-judged scoring instead.
Common pitfalls
- Applying exact match to a field that was never designed to have one correct phrasing — the fix is redesigning the field, not abandoning exact match everywhere.
- Treating EM and LLM-judged scores as interchangeable — they measure different things and can disagree substantially on the same answer, especially on borderline cases.