On DecisionSynth Bench's held-out set, Mem0's Claude-creator configuration retrieves the correct episode for a given question 78.1% of the time (R@5 0.781) — the strongest identity-recall score of any configuration tested for that vendor. On the rule-attribution task type specifically — which regulatory figure governed this decision — the same configuration scores 0.000 exact-match. Not low. Zero. Every retrieved memory for those tasks contained no citation string at all.
That pairing is the finding worth sitting with. This isn't a system that's generally unreliable and happens to fail at attribution along with everything else — its identity recall is the best in its own row. It's a system that specifically, consistently, discarded one field — the regulatory citation — while keeping the rest of what it needed to answer most other question types well.
What was actually lost
DecisionSynth Bench's episodes carry a semantic_refs field: dotted fact keys (like retirement.401k.elective_deferral.under_50) citing the specific regulatory figure that permitted or blocked each option a decision considered. A rule-attribution task asks which of those keys governed a given episode, with the answer key being an exact list of the correct fact strings. The Claude-creator Mem0 configuration's extraction step evidently didn't classify those citation strings as salient enough to keep — its retrieved memories for these tasks contain no reference strings whatsoever, not approximately-right ones or partially-complete lists.
Why this is the specific failure a regulated deployment can't accept
An advisor-facing or compliance-adjacent AI agent's job, when asked to justify a past recommendation, is not just "what did we recommend" but "what permitted us to recommend it." A system that can correctly answer the first question and not the second is not a system with a minor gap — it's a system that can produce a confident-sounding decision summary while being structurally unable to produce the one piece of evidence an examiner would ask for first. The identity-recall strength (R@5 0.781) makes this worse, not better, from a deployment-risk standpoint: a system that fails obviously (can't find the episode at all) gets caught in testing. A system that finds the right episode and answers everything except the citation looks reliable right up until someone asks the one question it can't answer.
| Task type | What it tests | Mem0 (claude-creator) held-out EM | |
|---|---|---|---|
| Direct recall | What was decided | 0.238 | |
| Rationale lookup | Why it was overridden | 0.010 | |
| Rule attribution | Which regulatory figure governed it | 0.000 |
Worth noting in the same row: this specific configuration's overall exact-match (0.136) and most other task-type columns are also weak, not just attribution — so this isn't a system that's excellent everywhere except one blind spot. It's a system with broad extraction loss, where attribution happens to be the column that hits zero. The retrieval strength (R@5 0.781) is the one number that stands out as genuinely strong against that backdrop, which is precisely what makes the citation loss legible as a specific, isolable failure rather than just "a weak configuration."
This is a configuration finding, not a product verdict
The same vendor's Gemini-creator configuration scores 0.470 attribution EM on the identical held-out task set — not perfect, but nowhere near zero. The difference between the two numbers is entirely the creating model, with the pipeline and task set held constant. That's strong evidence this specific failure is a property of the pipeline-model interaction under this configuration, not an inherent limitation of fact-extraction as an architecture, or of this vendor's product across every way it can be deployed.
What the fix would have to target
Because the paired Gemini-creator configuration (0.470 attribution EM) runs the identical pipeline and schema against the identical corpus, the difference isn't a missing capability in the product — the pipeline is capable of preserving citation strings, since it does so under one creator model and not the other. That points the likely fix at the extraction step's salience judgment specifically: whatever the pipeline's extraction prompt asks the Claude-creator configuration to prioritize, a dotted regulatory-citation string apparently doesn't clear the bar, while the same content clears it often enough under Gemini to retain nearly half. A targeted fix — explicitly enumerating citation fields as required extraction targets, rather than trusting general-purpose salience judgment to catch them — is a narrower intervention than switching creator models entirely, and one a team operating this pipeline in production could test directly against this same task type before deploying either configuration.
The practical takeaway for anyone building a citation-dependent agent
If an application's value depends on being able to reconstruct why a decision was permitted — not just what was decided — rule attribution needs to be tested explicitly and separately from general recall, because a system can pass every other check and still fail this one completely. The specific, quantified failure mode documented here (strong identity recall, zero citation recall, same configuration) is exactly the shape of error a spot-check conversation with an agent would very plausibly miss: ask it what happened, get a coherent answer; ask it what rule allowed that, and only then discover the gap.
Key takeaways
- Mem0's Claude-creator configuration scored attribution EM 0.000 on DecisionSynth Bench's held-out set — zero regulatory citations retrieved across the relevant task set — while scoring R@5 0.781 on identity recall in the same configuration.
- The failure is specific to the citation field, not a general extraction collapse presenting as one bad column: the pairing of strong identity recall with total citation loss is what makes it diagnosable.
- The same vendor's Gemini-creator configuration on the identical pipeline and corpus scores 0.470 attribution EM — evidence the failure is configuration-specific, not an inherent property of extraction-based memory as an architecture.
- Rule attribution needs to be tested explicitly for any citation-dependent deployment — a system that answers 'what happened' correctly can still fail 'what permitted it' completely, and that gap won't surface from casual testing.