Your memory eval is overfit to its corpus (extraction swings, verbatim doesn't)

DecisionSynth Bench runs every vendor system on two corpora: a 591-episode public dev set and a 543-episode private held-out set, both drawn from the same task design but over disjoint households. Mem0's Gemini-creator configuration scored 0.402 rationale-lookup exact-match on the dev set. On the held-out set — same pipeline, same task type, a different but comparably-sized sample of the same kind of data — it scored 0.038. Its overall exact-match rose over the same interval, from 0.321 to 0.465. One pipeline, one evaluation run each, two numbers that moved in opposite directions on the same underlying question.

That's not measurement noise at a scale worth dismissing — it's a roughly 10x collapse on one task-type column, while the system's headline number moved the other way. A team that ran this evaluation once, on one corpus, would have published a rationale-lookup score with no idea it was corpus-specific, and would have had no way to know whether the number they published was representative or a fluke of that particular sample.

What single-corpus evaluation actually measures

The uncomfortable fact about any benchmark score is that it measures performance on the corpus it was run against, and generalization beyond that corpus is an assumption, not a finding — unless the evaluation design tests generalization directly. A memory pipeline's extraction step, in particular, has plenty of surface area to fit incidentally to properties of one specific sample: the phrasing conventions of that corpus's decision episodes, the relative frequency of certain trigger types, the specific vocabulary its regulatory citations use. None of that is "cheating" in any deliberate sense — it's simply what happens when a system's behavior depends on patterns in the data it processes, and one corpus is one sample of those patterns.

The comparison that makes it a methodology finding, not a fluke

The reason this rises to a general claim about evaluation design — rather than a specific complaint about one Mem0 configuration — is the contrast with the deterministic verbatim baseline over the same dev-to-held-out transition. The baseline's overall exact-match moved from 0.950 to 0.980; its rationale-lookup column moved from 0.964 to 0.971. Both numbers went up, together, by a small amount consistent with the held-out set's slightly different task-type mix. No collapse, no divergence between the headline number and a specific column.

That's the actual comparison worth drawing: a system that does no extraction shows stable, correlated behavior across corpora, because there's no extraction step to have corpus-specific behavior in the first place. A system that extracts shows exactly the instability an extraction step would predict — and the instability doesn't even move in a single direction. Overall score up, one specific column collapsed. A single-number summary would have hidden the collapse entirely.

What multi-corpus evaluation catches that single-corpus can't

The practical argument isn't "run your evaluation twice for luck" — it's that a benchmark's held-out set exists specifically to answer a question a dev-set-only run structurally cannot: does this score generalize, or is it a property of this particular sample? Three things a single-corpus evaluation misses, that this comparison surfaces directly:

• Column-level instability hidden inside a stable-looking headline number. Mem0's overall EM improvement (0.321 → 0.465) would read as unambiguous progress if that were the only number reported. The rationale-lookup collapse underneath it says otherwise for that specific capability.
• Whether a pipeline's behavior is corpus-general or corpus-specific. The verbatim baseline's stability across corpora is itself informative — it's evidence that the task design is comparable across the two sets, which is what makes the extraction pipeline's instability attributable to the pipeline rather than to the held-out set being unfairly different.
• Which specific capability to distrust in production. A team relying on this pipeline for rationale-lookup specifically — "why was this decision made" — now has a concrete, quantified reason to distrust that capability's stability, which a single aggregate score would never have surfaced.

“

Single-corpus memory evals — including a single run of this one — overfit their corpus. That's an argument for benchmarks with held-out sets, not against this one, but we'd rather state it than have you find it.

Why the two corpora are actually independent

The comparison only means something if the dev and held-out sets are genuinely different data, not the same episodes reshuffled. DecisionSynth Bench's sampling design walks each archetype's seed frame in opposite directions for the two sets — the dev set draws ascending from the first half of each archetype's canonical seeds, the held-out set draws descending from the remaining half — with zero household overlap between them, asserted programmatically at build time rather than assumed. That's what makes the held-out run a genuine test of generalization rather than a second look at the same information: the households, and every episode generated from them, are disjoint by construction, not just by convention.

That same disjointness is also why this finding couldn't be explained away as the pipeline "remembering" something about the dev set and repeating it — there's nothing in the held-out households for a memory system to have seen before, whether through legitimate development iteration or accidental leakage. The instability documented here is the pipeline's extraction behavior genuinely differing across two samples of comparable decision episodes it has never processed before either time.

The honest limit of this finding

This is a single documented instance, on one pipeline, one task-type column, between two corpora. It is not a general claim that all extraction-based memory systems are unstable across corpora, or that DecisionSynth Bench's own held-out numbers are themselves immune to the same effect on some other axis — a genuinely honest evaluation methodology has to apply its own skepticism to itself, not just to the systems it scores. What it does establish, concretely, is that the possibility of this kind of collapse is real and large enough to matter, and that a benchmark design without a held-out set has no mechanism to catch it when it happens.