The first row of DecisionSynth Bench's scoreboard is a system with no machine learning in it at all: store every episode as text, retrieve the ones that share the most keywords with the question, pull the answer field directly out of whatever it finds. On the 543-episode held-out set, that system scores 0.980 overall exact-match. Every vendor system tested against the same set — Letta, Mem0, Zep/Graphiti — scores lower, several of them far lower. A reader's first reaction is reasonably "if grep nearly wins, what is this benchmark actually testing?" The answer is specific enough to state directly: it's testing how much decision-relevant structure survives your system's compression, and the near-ceiling baseline is the number the rest of the scoreboard is measured against, not a symptom of the benchmark being too easy.
Why the baseline is supposed to score this high
Production memory systems don't store everything verbatim, because verbatim storage doesn't scale — the whole reason to extract facts, summarize, or build a graph is to avoid keeping every raw record forever in a form expensive to search. That means every production system is making a bet: that whatever it keeps after compression is enough to answer the questions that matter. A benchmark can only tell you whether that bet paid off if it has a reference point for what "everything" would have scored — which is exactly what a system that keeps everything and does no compression provides.
At a corpus this size — 543 held-out episodes, 1,367 tasks — verbatim storage and keyword retrieval is a genuinely near-ceiling strategy: there's little the deterministic baseline could plausibly miss, since it never throws anything away. That's the point, not an oversight. The gap between any vendor's row and this one is a direct, uncontaminated measurement of what that vendor's compression pipeline cost, because the only variable that changed between the two rows is how much structure survived the write step.
| System | What it discards | Overall EM (held-out) | |
|---|---|---|---|
| Naive-lexical baseline | Nothing — stores every episode verbatim | 0.980 | |
| Letta (archival) | Almost nothing — near-verbatim passages | 0.917–0.922 | |
| Mem0 (extraction) | Most field detail — keeps discrete facts only | 0.136–0.465 | |
| Zep/Graphiti (graph) | Record cohesion — keeps fine-grained edges | 0.091–0.215 |
The number that isn't near-ceiling, and why
The baseline's precision@5 (P@5) is 0.302 — nowhere close to 1.0, and a reader tallying scores might read that as the baseline failing somewhere. It isn't. P@5 for any task with exactly one relevant piece of evidence is mathematically bounded at 0.2 by construction: one relevant item out of five returned, regardless of how well the system retrieves. Most of DecisionSynth Bench's task types (direct recall, rationale lookup, rule attribution) have exactly one relevant episode as evidence, so a P@5 near 0.2 on those tasks isn't a retrieval failure — it's the ceiling the task shape imposes on any system, including a perfect one. The blended 0.302 sits close to that structural floor precisely because the baseline is retrieving well; a lower P@5 blended average would actually indicate a problem, and a much higher one would be a sign the task mix skewed toward multi-evidence questions, not that the system improved.
What "the benchmark measures compression loss, not retrieval IQ" actually means
A benchmark where the simplest strategy nearly maxes out the score is not measuring which system is cleverest at reasoning over the corpus — a corpus this size doesn't require cleverness to search exhaustively. It's measuring something narrower and, for this specific evaluation question, more useful: given that production systems have to compress, how much of what matters survives the compression each one chose. That framing explains why the scoreboard's other rows spread out so widely below the baseline (0.980 down to 0.091) even though none of the underlying task difficulty changed between rows — what changed is how much structure each system's pipeline kept.
The naive-lexical baseline stores every episode verbatim and greps it — a near-ceiling strategy at this corpus size, and deliberately so. Production agent-memory systems do not store verbatim; they extract, summarize, and graph. The measurement is how much decision-relevant structure survives your system's compression.
Reading any scoreboard row against the baseline, in practice
The baseline's job is to be the first number you look at, not the last. Concretely, that means checking three things before drawing a conclusion from any vendor row on this scoreboard:
- The overall gap. Letta's held-out overall EM (0.917–0.922) sits 0.06–0.06 below the baseline's 0.980 — a small gap, meaning its archival, near-verbatim design lost comparatively little. Mem0's best row (0.465) and Graphiti's best row (0.215) sit 0.5+ and 0.7+ below it respectively — large gaps, meaning their compression steps discarded much more.
- Which specific columns carry the gap. A vendor's overall gap to baseline is rarely uniform across task types — Graphiti's gap is narrow on temporal ordering and enormous on rule attribution, for instance. The aggregate number alone doesn't show you where the loss concentrated.
- Whether the metric itself is structurally bounded. Precision@5's 0.2 ceiling on single-evidence tasks is one example; any benchmark can have others. A metric that looks "bad" relative to a naive expectation of 1.0 may simply be behaving exactly as the task design predicts.
Skipping any of these three and reading only the headline overall-EM column is how a benchmark table gets misread — either by treating the near-ceiling baseline as an indictment of the benchmark, or by treating a vendor's low overall score as a uniform failure rather than a specific, diagnosable one.
Two numbers that corroborate this isn't a fluke of one run
The same near-ceiling pattern shows up on both of DecisionSynth Bench's corpora, not just the held-out set: the public dev set's baseline scores 0.950 overall EM (P@5 0.292, R@5 0.972) — close to, though not identical to, the held-out figures (0.980, 0.302, 0.994). The two runs use different episodes over disjoint households, so exact agreement isn't expected, but the consistency of the pattern — a verbatim strategy landing near-ceiling on both independently generated corpora — is itself evidence the design does what it's meant to do, rather than being an artifact of one particular sample.
Key takeaways
- The deterministic verbatim baseline scores 0.980 overall EM on the held-out set by design — it's the ceiling reference every other score is measured against, not evidence the benchmark is too easy.
- The gap between a vendor's score and the baseline's score is a direct measurement of what that vendor's compression pipeline cost, because verbatim storage is the one strategy that discards nothing.
- The baseline's precision@5 (0.302) looks low only until you account for the structural cap of 0.2 on single-evidence tasks — recall@5 (0.994) is the number that actually reflects its retrieval quality.
- The same near-ceiling pattern holds on both the public dev set (0.950 EM) and the private held-out set (0.980 EM), over disjoint households — evidence the pattern is a property of the design, not one sample.