A scoreboard row is pipeline × model: the case for publishing both creators

Most vendor memory-system benchmarks publish one score per product. DecisionSynth Bench publishes two rows per vendor — every configuration runs once with gemini-2.5-flash and once with claude-haiku-4-5 as the model performing extraction and answer generation, with embeddings (gemini-embedding-001) held constant across both runs. The reason isn't thoroughness for its own sake. It's that a vendor pipeline embeds a language model, which means a "vendor score" is actually a measurement of pipeline and model together — and on this benchmark's evidence, collapsing that into one number would have hidden which of the two variables was doing the work.

What one row would have hidden

Take Letta and Mem0 side by side. Letta's held-out overall EM moves from 0.922 (Gemini-creator) to 0.917 (Claude-creator) — a difference of 0.005, functionally noise. Mem0's moves from 0.465 (Gemini-creator) to 0.136 (Claude-creator) — a difference of 0.329, larger than the gap between some entirely different vendors in the scoreboard.

A single published row per vendor — whichever creator model the benchmark's maintainers happened to pick — would have reported one of these numbers as "Mem0's score" with no indication that the other creator model produces a result 2.4x lower on the same held-out corpus. Anyone using that single number to decide between vendors would be, unknowingly, also deciding between creator models without being told the choice existed.

Why the swings go in different directions, not just different amounts

The two-creator design surfaces something a magnitude comparison alone wouldn't: the swings aren't just larger or smaller across systems, they trade off differently structured losses. Under Mem0, the Claude-creator configuration preserves episode identity substantially better than Gemini (held-out R@5 0.781 vs. 0.439) but discards regulatory-citation detail entirely (attribution EM 0.000 vs. 0.470) — a near-total swap of which structure survives. Under Zep/Graphiti, Claude-haiku extracts roughly 2.8 edge facts per episode against Gemini-flash's roughly 18, starving the graph and costing both retrieval (R@5 0.425 vs. 0.737) and temporal ordering (EM 0.389 vs. 0.886) in the same direction rather than trading one structure for another.

That distinction matters for anyone trying to generalize from a benchmark score to their own deployment. "Mem0's Claude configuration loses citations" and "Mem0's Gemini configuration loses identity" are two different engineering problems requiring two different mitigations, and a single blended or averaged score would present neither problem clearly — it would just look like a medium score with no actionable read on what specifically to fix.

The cross-validator: a second use for the second model

Running two creator models also enabled a validation method that a single-model benchmark can't perform: each family's answers are audited by the other family, working from the same retrieved context, with the validator never shown ground truth. If Gemini-creator produced an answer, Claude checks it against the retrieved evidence (and vice versa) — a health check that requires nothing except two independent model families already being in the pipeline for other reasons.

Disagreement rates order the six rows almost exactly the way exact-match does — Letta's two rows show the lowest disagreement (0.087, 0.129), Mem0's sit in the middle (0.386, 0.492), and Zep/Graphiti's Claude-creator row shows the highest of any configuration (0.908). That correlation is the actual payoff of the two-creator design: it demonstrates that cross-model disagreement tracks answer quality even when no ground truth is available to check against directly — which is exactly the situation any team monitoring a memory system in production is in, since production questions don't come with an answer key.

What single-model vendor evals miss

The general claim this design supports: any memory-benchmark score that doesn't disclose which model produced it is withholding a variable capable of swinging the result by more than 2x in either direction — sometimes trading one kind of structural loss for a completely different one, not just a better or worse version of the same loss. A benchmark methodology that runs one model per vendor can't detect this, can't report it, and — worse — presents a single number with the same apparent authority as a benchmark that tested the variable and found it stable.

The full disagreement table, not just the range

The cross-validator disagreement rate — how often the auditing model family flags the answering family's response as wrong, with neither seeing ground truth — is published for all six configurations, not just the two extremes:

Two things stand out in the full table that the range alone doesn't show. First, the ordering is stable within each vendor regardless of which family is doing the auditing — Letta's two rows cluster low, Mem0's sit in the middle, Graphiti's sit high, no matter which model is checking which. Second, the ordering across vendors tracks exact-match ordering almost exactly (Letta lowest disagreement and highest EM, Graphiti highest disagreement and lowest EM), which is the actual evidence for treating disagreement as a quality proxy — it isn't just plausible in principle, it reproduces the ranking a ground-truth-based metric independently produced, on the same six rows, without ever seeing the answer key.