Mem0 vs. Zep vs. Letta: What a Decision-Recall Benchmark Actually Shows
DecisionSynth Bench's maintainer-run scoreboard puts three production memory systems — Mem0, Zep/Graphiti, and Letta — through the same 543 held-out decision episodes (1,367 QA tasks), over households none of them had seen. Every configuration runs twice, once with `gemini-2.5-flash` and once with `claude-haiku-4-5` as the creating model, so a vendor's score is really pipeline × extraction model, not a single number. The spread across the six resulting rows is the finding: verbatim storage (Letta) sits at 0.917–0.922 overall exact-match, within a few points of the deterministic no-LLM baseline's 0.980. The extraction and graph pipelines (Mem0, Zep/Graphiti) sit far below that — 0.465 and 0.215 at their best configurations — because they compress what they store, and compression is lossy in specific, measurable ways.
The two options
Letta (archival — stores passages verbatim, no extraction)
Letta's archival memory stores retrieved passages close to verbatim rather than extracting discrete facts. On the held-out set it scores 0.922 overall EM (gemini-creator) and 0.917 (claude-creator), both within 0.06 of the 0.980 no-LLM baseline.
- Creator-invariant: 0.922 (gemini) vs. 0.917 (claude) — because nothing is extracted, the choice of creating LLM barely moves the score, unlike the other two systems
- Lowest cross-validator disagreement of any vendor rows in the scoreboard: 0.129 (gemini-creator, audited by Claude) and 0.087 (claude-creator, audited by Gemini) — the two lowest of all six vendor×creator rows
- Retrieval (R@5 0.821) and per-task scores stay close together across task types — recall EM 0.947, rationale EM 0.905, attribution EM 0.956 — no single task type collapses
- Still not the baseline — R@5 0.821 vs. the baseline's 0.994 means some retrieval quality is lost even without an extraction step
- Its weakest column is precedent search at EM 0.667 — lower than every other column for this system, on a task type that requires aggregating across multiple episodes rather than reading one closely
- Storing near-verbatim content doesn't scale the way a compressed representation does — the same property that keeps its score high is a cost tradeoff in production, not a free win
Choose an archival, low-compression approach when near-complete fidelity matters more than the storage and retrieval efficiency that extraction or graphing would buy — and when the corpus size stays small enough that verbatim storage remains practical.
Mem0 and Zep/Graphiti (extraction & graph pipelines — compress before storing)
Both systems process raw content into a compressed representation before storing it — Mem0 extracts discrete facts, Graphiti builds a temporal knowledge graph. Both score well below Letta on overall exact-match, and each loses a different, specific kind of structure.
- This is the entire design point: extraction and graphing are what let a memory system scale past keeping everything verbatim forever
- Zep/Graphiti's temporal ordering is the one column where an LLM-backed system beats everything except the deterministic baseline: EM 0.886 (gemini-creator), ahead of Letta's 0.760 — the temporal graph doing exactly the job it was built for
- Mem0 (gemini-creator) is the best-scoring extraction pipeline overall at EM 0.465 — meaningfully ahead of Graphiti's best row
- Mem0 (claude-creator) posts attribution EM 0.000 on the held-out set — every retrieved memory contains zero regulatory-citation strings, a total loss of exactly the field a compliance reviewer would ask for first
- Mem0 (gemini-creator)'s rationale EM collapsed from 0.402 on the dev set to 0.038 on held-out, while its overall EM rose (0.321 → 0.465) — the same pipeline preserves different structure on different data, a sign of corpus-specific overfitting
- Graphiti's retrieval unit is the individual edge fact, and one decision memo shreds into roughly 25 of them — at the matched 5-item retrieval budget, a system can win temporal ordering and still fail exact-identifier recall (EM 0.110) because the budget is spent on fragments of the record it needs, not the whole thing
- Graphiti's cross-validator disagreement under the Claude-creator variant reaches 0.908 — the highest of any row in the scoreboard, meaning two independent model families agree on its answers barely 9% of the time
Choose an extraction or graph pipeline when raw verbatim storage won't scale for your corpus size, and specifically choose the one whose preserved structure matches your use case: Mem0 if individual-fact recall matters most and you can tolerate citation loss under some configurations; Graphiti if temporal ordering across decisions is the question you need answered and exact-identifier lookups can go through a different path.
Decision framework
The scoreboard's own framing is the right lens: the deterministic baseline stores every episode verbatim and greps it, which is a near-ceiling strategy at this corpus size and deliberately so. The gap between any vendor's row and that baseline is what its pipeline's compression cost. Letta's small gap says it compresses least. Mem0 and Graphiti's large gaps say they compress a lot — which is the point of building them, not a defect — but the two lose different structure in the process, and neither loses it uniformly.
The two-creator design (every configuration run once with Gemini, once with Claude) exists because a 'vendor score' that reports only one creating model is hiding a variable as large as the vendor choice itself. Letta barely moves across creators because it doesn't extract anything for a creator model to shape. Mem0 and Graphiti swing hard, and in opposite directions: under Mem0, Claude-creator preserves episode identity far better than Gemini (held-out R@5 0.781 vs. 0.439) but discards citation detail entirely (attribution EM 0.000); under Graphiti, Claude-haiku extracts roughly 2.8 edge facts per episode where Gemini-flash extracts roughly 18, starving the graph and costing both retrieval (R@5 0.425 vs. 0.737) and temporal EM (0.389 vs. 0.886). The same underlying model loses different things depending on which pipeline it's embedded in — which is a reason to distrust any benchmark that publishes one row per vendor.
For a team choosing between these systems, the practical question isn't 'which one wins' — none of them wins uniformly — but 'which specific structure does my application need to survive compression,' checked against the column that tests exactly that.
Bottom line
There is no single winner among Mem0, Zep/Graphiti, and Letta on this benchmark — there's a verbatim-storage system that loses the least because it compresses the least, and two compression pipelines that each trade a specific kind of structure for scale, in ways that swing with the creating model as much as with the vendor. A team picking one of these for a decision-relevant memory system should run the adapter against the task types its application actually needs — rationale lookup, rule attribution, temporal ordering — rather than reading a single overall number.
FAQ
Which system scored highest overall on the held-out set?+
Letta (archival), at 0.922 overall EM with the Gemini creator and 0.917 with Claude — both close to the 0.980 deterministic baseline because its archival design does no extraction. Among the two compression pipelines, Mem0's Gemini-creator configuration scored highest at 0.465.
Why does Zep/Graphiti score lowest overall but win temporal ordering?+
Its retrieval unit — an individual edge fact rather than a passage or a consolidated record — is exactly the structure a temporal-ordering question needs (each edge carries its own validity window), which is why its temporal EM (0.886) beats every other LLM-backed system. The same fine-grained unit works against it on tasks needing a complete record: one decision memo shreds into roughly 25 facts, so a fixed 5-item retrieval budget often can't surface a complete picture, which is why its exact-identifier recall EM is only 0.110.
What does 'attribution EM 0.000' mean specifically?+
It means that in the Mem0 claude-creator configuration, none of the retrieved memories for the relevant held-out tasks contained a regulatory-citation string (a `semantic_refs`-style dotted fact key) at all — not a wrong citation, but no citation. The same configuration's identity recall (R@5 0.781) was comparatively strong, showing the extraction step preserved which episode mattered while discarding the specific field a compliance answer needs.
Is a lower score evidence that a system is bad?+
Not on its own — it's evidence of what that system's design trades away for the ability to scale past verbatim storage. A system's fitness depends on which structure your use case actually needs retrieved; the per-task-type columns, not the overall EM, are where that answer lives.
Can I reproduce these numbers myself?+
The dev-set scores (a separate 591-episode public set with full answer keys) are reproducible from a clean clone with zero API keys for the baseline, and with your own API keys for the vendor adapters (`bench/adapters/`). The held-out numbers above are maintainer-run on a private set whose episodes and answers are never published — that's the benchmark's structural defense against a system imitating its way to a good score.
How were these vendor rows configured?+
All three ran on an identical model stack per creator variant (`gemini-2.5-flash` / `claude-haiku-4-5` for extraction and answer generation, `gemini-embedding-001` for embeddings throughout) and a matched 25-item retrieval budget in each system's native unit. Exact per-vendor configuration is documented in each adapter's header in the repo.