Agent Memory Benchmarks in 2026: The Complete List
"Agent memory" covers more failure modes than any single benchmark tests. As of 2026, four public benchmarks between them cover the ground: LoCoMo tests whether a fact stated early in a long conversation gets retrieved later. LongMemEval adds knowledge updates and abstention to that picture — does the system know when a fact went stale, and does it know when to say it doesn't know. BEAM tests whether raw context length can substitute for memory at all, at scale up to 10M tokens. And DecisionSynth Bench tests a category none of the first three touch: whether a memory system can retrieve the structure of a decision — what was chosen, why, and which rule permitted it — rather than just a fact that was stated. This is the complete list of that quartet, what each one actually measures, and where the coverage still has gaps.
The two options
DecisionSynth Bench
The only benchmark in the group testing decision-shaped retrieval rather than conversational recall: direct recall, rationale lookup, precedent search, temporal ordering, and rule attribution over synthetic financial-advisor decision episodes, with answer keys emitted by the generator at creation time.
- Five task types with no equivalent elsewhere in this list: direct_recall (what was decided), rationale_lookup (why it was overridden), precedent_search (which similar episodes overrode for the same reason), temporal_ordering (what preceded a decision), and rule_attribution (which cited regulatory figure governed the choice)
- Known-answer by construction — no hand-labeling, no LLM-judged labels; every answer key is a typed field the generator wrote when it created the episode
- Schema and harness are Apache-2.0; the ~591-episode / 1,487-task dev set ships CC BY 4.0 with full answer keys for free development
- A private held-out set (disjoint households, answers never published) runs the actual public scoreboard — a structural defense against a system imitating its way to a good score on the free sample
- Domain-specific — financial-services advisor decisions over synthetic households, not a general-purpose chat-memory test
- No abstention category — the deterministic generation path can't construct the kind of genuinely ambiguous 'should the system say it doesn't know' question a human annotator can design on purpose
- Newest of the four — less citation history than LoCoMo, though the same is true of BEAM and LongMemEval relative to LoCoMo
Reach for DecisionSynth Bench when the system under evaluation sits between a client and a decision with real structure — an advisor agent, a compliance-adjacent assistant, anything where 'what did we decide and why' is a question the system needs to answer correctly, not just 'what did the client say.'
The established conversational-memory benchmarks (LoCoMo, LongMemEval, BEAM)
Three benchmarks that, between them, cover general chat-memory recall, sustained-session nuance including abstention and knowledge updates, and raw scale — each widely used, each licensed and structured differently.
- LoCoMo (Maharana et al., ACL 2024) — conversations averaging 600 turns / 16K tokens over up to 32 sessions; QA split into single-hop, multi-hop, temporal, and open-domain categories; released CC BY-NC 4.0 (non-commercial). The most-cited reference point in the field, and the benchmark most other memory-system papers, including Mem0's own, report scores against.
- LongMemEval (Wu et al., ICLR 2025) — 500 questions across five memory abilities (information extraction, multi-session reasoning, temporal reasoning, knowledge updates, abstention); ground truth is human-authored (LLM-seeded, then manually filtered, rewritten, and decomposed by annotators); released CC BY 4.0. Commercial assistants show roughly a 30% accuracy drop under sustained interaction on this set.
- BEAM (Tavakoli et al., ICLR 2026) — 100 conversations / 2,000 validated questions, generated up to 10M tokens; built specifically to test whether a large context window makes memory systems unnecessary. Finding: it doesn't — even 1M-token-context models degrade as dialogues lengthen. Benchmark code MIT-licensed; data CC BY-SA 4.0.
- All three are general-purpose — applicable to any conversational domain, not tied to a specific vertical
- None of the three test decision structure — no rationale, no override, no precedent, no rule citation; a system can score well on all three and still be unable to explain why a decision was made
- LoCoMo's license (CC BY-NC 4.0) rules out commercial use of the dataset itself, not just reporting a score against it
- LoCoMo has no private held-out split — answers ship with the public data
Run these three when the system is a general-purpose conversational agent. LongMemEval if abstention and knowledge-currency matter. BEAM if the deployment genuinely pushes context length (very long sessions, very large corpora in context). LoCoMo as the baseline recall check most other published scores can be compared against.
Decision framework
None of the four is a superset of the others, so 'which one should we run' is the wrong framing — the right one is 'which of these four gaps does our system actually need to close.' A general-purpose assistant with no decision-shaped output only needs the top three. An advisor-facing or compliance-adjacent agent needs DecisionSynth Bench specifically, because none of the conversational benchmarks will ever surface a rule-attribution failure — there's no rule to attribute in a persona-grounded chat dataset.
Teams with the engineering budget to run more than one get a more complete picture than any single benchmark provides: LoCoMo and LongMemEval catch basic recall and sustained-session regressions; BEAM catches whether the system quietly started relying on context-window size instead of a real memory pipeline; DecisionSynth Bench catches whether decision-relevant structure survives whatever compression the memory pipeline applies. A system that passes all four has had four different kinds of forgetting tested, not one.
Bottom line
As of 2026, LoCoMo, LongMemEval, and BEAM cover conversational memory thoroughly — recall, nuance, and scale. DecisionSynth Bench is the one addition that tests something structurally different: not what was said, but what was decided, why, and under which rule. A memory-evaluation program that only runs the conversational three has not tested whether its system can support an advisor agent under compliance review, because none of those three ask that question.
FAQ
What's the single biggest difference between DecisionSynth Bench and the other three?+
Task shape. LoCoMo, LongMemEval, and BEAM all test recall or reasoning over conversational content — facts a person said, at some point, in a chat. DecisionSynth Bench tests retrieval of decision structure — trigger, options considered, recommendation, override reason, outcome, and the cited rule — which doesn't exist as a concept in a conversational-recall dataset.
Which of these benchmarks are free to use?+
All four publish a dataset and code. Licenses differ: LoCoMo's dataset is CC BY-NC 4.0 (research/non-commercial use); LongMemEval is CC BY 4.0; BEAM's benchmark code is MIT and its data CC BY-SA 4.0; DecisionSynth Bench's schema and harness are Apache-2.0 and its public dev-set corpus is CC BY 4.0, with the private held-out set never published (it runs the maintainer-operated scoreboard).
Is there a benchmark that combines all four axes into one test?+
Not as of 2026. Each was built by a different team to answer a different question, and combining them into a single scored benchmark would require reconciling four different corpus designs, task-answer formats, and scoring conventions. Running them independently and comparing the resulting profile — strong on recall, weak on rule attribution, for instance — is the current state of the art.
How does DecisionSynth Bench defend against a system gaming the public dev set?+
The public scoreboard runs on a private held-out set of episodes over households the dev set never includes, asserted disjoint at build time. A system (or a model) that has memorized or pattern-matched the free dev set still has to produce verified-correct answers against households it has never seen, which a lookup or imitation strategy can't do.