LoCoMo Alternatives: Memory Benchmarks That Test More Than Chat Recall
LoCoMo (Maharana et al., ACL 2024) is the benchmark most agent-memory teams cite first, and for a specific reason: it was among the first to test recall over genuinely long, multi-session dialogue rather than a single chat window. Its dataset averages 600 turns and 16K tokens per conversation, spread across up to 32 sessions, with questions split into single-hop, multi-hop, temporal-reasoning, and open-domain categories. Mem0's own evaluation paper reports its results on LoCoMo, which is part of why the name shows up in almost every agent-memory comparison since. But LoCoMo measures one thing: whether a fact stated in an earlier session gets retrieved correctly later. It was built to test conversational recall, and it tests conversational recall well. It was not built to test whether a system can explain a decision, find precedent, or attribute an outcome to the rule that governed it — and teams building memory for anything more structured than a chatbot eventually notice the gap. Three newer benchmarks fill different parts of it.
The two options
LoCoMo
A conversational-memory benchmark built around long, persona-grounded dialogues generated by a machine-human pipeline and verified by human annotators for consistency. Tests whether an agent's memory surfaces the right fact from earlier in a long conversation.
- Realistic long-dialogue construction — personas and temporal event graphs generate the conversations, and human annotators edit them for long-range consistency before release
- Established reference point — widely cited, and the benchmark most memory-system papers (including Mem0's own) report scores against
- Covers four distinct QA categories (single-hop, multi-hop, temporal, open-domain), not just flat fact lookup
- Includes multi-modal turns (agents share and react to images), which most text-only benchmarks skip
- Tests recall of stated facts, not the reasoning behind a decision — there is no override, no rationale, no rule citation to retrieve
- General-purpose personas (hobbies, relationships, opinions), not a regulated or specialized domain — nothing in the QA set resembles 'why was this policy overridden'
- Licensed CC BY-NC 4.0 (non-commercial) — a real constraint for any team that wants to build on the dataset itself, not just report a score against it
- No held-out/private test split — the full dataset with answers ships publicly, so a system's real-world generalization beyond the published set is untested
Choose LoCoMo-style evaluation when the system under test is a general-purpose conversational assistant and the question is narrow: does it remember what the user told it, several sessions ago, without re-asking. It is the right first check for any memory pipeline, and a reasonable one to run even if a more specialized benchmark also applies.
Decision-relevant alternatives (LongMemEval, BEAM, DecisionSynth Bench)
Three newer benchmarks, each closing a different gap LoCoMo leaves open: nuanced multi-session reasoning and abstention (LongMemEval), raw long-context stress-testing (BEAM), and decision-shaped retrieval with rule attribution (DecisionSynth Bench).
- LongMemEval (Wu et al., ICLR 2025) adds knowledge updates and abstention as first-class tests — 500 questions across five memory abilities, with commercial assistants showing roughly a 30% accuracy drop under sustained interaction
- BEAM (Tavakoli et al., ICLR 2026) tests memory at a scale LoCoMo doesn't approach — 100 conversations and 2,000 validated questions generated up to 10M tokens, built specifically to check whether a huge context window makes a memory system unnecessary (its finding: no, models still degrade as dialogues lengthen)
- DecisionSynth Bench tests a category none of the others touch: decision recall, rationale lookup, precedent search, temporal ordering of decisions, and which regulatory figure governed a choice — over synthetic financial-services households where every answer is emitted by the generator at creation time, not hand-labeled afterward
- Between them, the three cover knowledge currency, raw scale, and decision structure — the three axes LoCoMo's single-conversation recall test doesn't reach
- Less established than LoCoMo simply by age — LongMemEval and BEAM are newer, and citation volume takes time to catch up to a benchmark's actual coverage
- DecisionSynth Bench is domain-specific by design (financial-services advisor decisions) — it is not a general chat-memory benchmark and doesn't claim to be one
- Running more than one benchmark costs more engineering time than running one — each has its own harness and adapter contract
Pick based on the gap you actually need closed. Testing whether a system correctly says 'I don't know' or updates a stale fact: LongMemEval. Testing whether memory holds up at extreme context length: BEAM. Testing whether a system can retrieve the reasoning, the precedent, or the rule behind a decision — the shape of failure that matters for an advisor-facing or compliance-facing agent: DecisionSynth Bench.
Decision framework
Start from what your agent is actually for. A general-purpose assistant that just needs to not repeat itself across a long conversation is well served by LoCoMo-style evaluation — it's the most direct test of that specific failure mode, and the one most other systems already report scores against, which makes comparison easy.
The moment the agent's job involves decisions with structure — an option that was chosen over alternatives, a policy that was followed or overridden and why, a rule that permitted or blocked a choice — conversational recall stops being the binding constraint. A system can pass a recall benchmark by retrieving the sentence where a fact was stated, and still fail the question that actually matters to a compliance reviewer: not 'what did the client say,' but 'what did we decide, and why.' That gap is exactly where LoCoMo's QA categories (single-hop, multi-hop, temporal, open-domain fact lookup) stop and DecisionSynth Bench's task types (direct recall, rationale lookup, precedent search, temporal ordering, rule attribution) begin.
Most rigorous evaluation programs don't pick one. They run a conversational-recall benchmark to catch basic retrieval regressions cheaply, and a decision-shaped or domain-specific benchmark to catch the failures that actually show up in production for their use case.
Bottom line
LoCoMo is the right benchmark for testing whether a memory system remembers what was said. It was never built to test whether a system remembers what was decided, and extending it to do so would mean building a different benchmark — which is what DecisionSynth Bench is. If your agent sits between a client and a regulated decision, decision-shaped evaluation isn't a nice-to-have layered on top of conversational recall; it's testing the failure mode that actually reaches an examiner.
FAQ
What does LoCoMo actually test?+
Whether an agent's memory correctly retrieves a fact stated earlier in a long, multi-session conversation. The dataset averages 600 turns and 16K tokens per conversation over up to 32 sessions, with questions split into single-hop, multi-hop, temporal-reasoning, and open-domain categories (Maharana et al., ACL 2024).
Is DecisionSynth Bench a replacement for LoCoMo?+
No — they test different things. LoCoMo tests conversational recall over persona-grounded chat. DecisionSynth Bench tests retrieval of decision structure (what was decided, why, what rule governed it) over synthetic financial-advisor episodes. A system can score well on one and poorly on the other, because they're measuring different failure modes.
What is BEAM testing that LoCoMo doesn't?+
Scale, and a specific hypothesis: that a large enough context window makes memory systems unnecessary. BEAM generates conversations up to 10M tokens with 2,000 validated questions and finds that even 1M-token-context models degrade as dialogues lengthen — the context-window shortcut doesn't hold (Tavakoli et al., ICLR 2026).
How big is LongMemEval, and how were its answers labeled?+
500 questions spanning five memory abilities — information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. Questions were LLM-seeded and then manually filtered and rewritten by human annotators, who also manually decomposed the ground-truth answers into evidence statements (Wu et al., ICLR 2025, §3.2). That hand-authored ground truth is the direct contrast with DecisionSynth Bench's generator-emitted answer keys — see the dedicated comparison for that tradeoff.
Can I run more than one of these benchmarks on the same system?+
Yes, and it's the more informative approach if the engineering time is available. Each ships its own harness, so running all four means implementing (or reusing) an adapter per benchmark rather than one shared contract. DecisionSynth Bench's adapter is a single ingest/answer interface documented in the repo's bench/types.ts, with a zero-API-key deterministic baseline for a first run.