How to Evaluate AI Agent Memory: A Step-by-Step Method

Step 1: Get a corpus with a real answer key

Everything downstream depends on this step being solid. A QA pair is only useful for evaluation if the correct answer is knowable independent of the system being tested — otherwise you're not scoring the memory system, you're scoring your own judgment about whether its answer sounds plausible.

There are two legitimate ways to get that answer key. The first is hand-labeling: a human reads the source material and writes down the correct answer, ideally after a separate LLM-seeding pass to generate candidates for the human to filter and refine (this is LongMemEval's approach — 500 questions, LLM-seeded then manually rewritten and decomposed into evidence statements by annotators). The second is generation with the answer built in: a deterministic process creates the scenario and emits the ground truth in the same pass, so there's no separate labeling step to get wrong (this is DecisionSynth Bench's approach — every episode's outcome, override reason, and cited rule are typed fields the generator writes, not a human's after-the-fact interpretation).

Either is defensible. What isn't defensible is skipping this step — using an LLM to generate both the question and a 'gold' answer with no independent verification, then scoring another LLM's response against that ungrounded answer. That measures agreement between two models, not correctness.

Step 2: Ingest through an adapter that never sees the answers

The system under test needs a single, narrow contract: ingest the corpus once, then answer each question using only what it retrieved from its own memory. The ground truth — the answer key, the evidence ids, any distractors — must never reach the system being scored. This sounds obvious until you're debugging a harness at 11pm and it's tempting to pass a little extra context through 'just for this run.'

The adapter pattern keeps that boundary enforced in code, not in discipline. DecisionSynth Bench's harness defines it as two methods: one to ingest the corpus, one to answer a single task, with the runner stripping the answer key and evidence ids before the task ever reaches the adapter.

interface MemoryAdapter {
  name: string;
  // Called once with the full corpus before any questions.
  ingest(episodes: EpisodeRecord[]): Promise<void> | void;
  // Ground truth is NEVER passed in — the runner strips it first.
  answer(task: QaTaskWithoutAnswers): Promise<SystemAnswer> | SystemAnswer;
}

interface SystemAnswer {
  task_id: string;
  evidence_ids: string[];   // what the system retrieved, best-first
  answer_key: Record<string, unknown>; // typed answer, same shape as the real key
}

Step 3: Score two axes, not one

A memory system can fail in two independent ways: it can retrieve the wrong evidence, or it can retrieve the right evidence and still answer wrong. Collapsing both into a single pass/fail score hides which failure you're looking at, so the method scores them separately and reports both.

Retrieval is scored as precision and recall at k — of the evidence the system returned, how much was actually relevant (precision), and of the evidence that was relevant, how much did the system find (recall). Answer correctness is scored as exact match against the typed ground-truth key: an unordered set match for a question like 'which episodes overrode for this reason,' an ordered match for 'what happened first, second, third,' and a field-wise match everywhere else. Exact match on a typed field, not a free-text similarity score, is the point — 'approximately right' is not a category a compliance reviewer accepts, so the scoring shouldn't accept it either.

The two axes matter because they diagnose differently. A system with high recall and low exact-match retrieved the right material and still got the answer wrong — a reasoning or extraction problem downstream of retrieval. A system with low recall and (occasionally) correct answers got lucky, or is answering from something other than what it actually retrieved — worth knowing before you trust it in production.

Step 4: Read the score against a baseline, not in isolation

A raw score means little without a reference point. The reference point that matters most is a baseline that stores everything verbatim and retrieves by simple lexical overlap — no extraction, no summarization, no graph, just grep. That baseline isn't a strawman; it's the ceiling. Production memory systems compress by design (they extract facts, summarize, or build a graph, because storing every raw record forever doesn't scale), and compression is lossy. The gap between a system's score and the verbatim baseline's score is a direct measurement of what that system's compression lost.

On DecisionSynth Bench's public dev set (591 episodes, 1,487 QA tasks), the committed verbatim baseline scores 0.950 overall exact-match, 0.292 precision@5, and 0.972 recall@5. That precision number looks low next to the other two until you read the footnote: for a single-evidence task, precision@5 is mathematically capped at 0.2 (one relevant id out of five returned), so 0.292 overall is actually close to the ceiling once the multi-evidence task types are folded in. Reading a benchmark table means checking what's structurally bounded before treating a number as a weakness.

Fairness rules that make a score defensible

A benchmark score only means something if someone else can reproduce it, or at least audit how it was produced. The rules that keep a scoreboard honest are simple and worth stating explicitly rather than assuming: no ground truth reaches the adapter under any circumstance, every system gets the same retrieval budget, the exact configuration (model names, index settings) needed to reproduce a run ships alongside the score, and published rows come from reruns on a held-out set the system couldn't have seen during development — not from whatever number a vendor self-reports.

That last rule is why a public dev set with answers and a private held-out set without them is a standard pattern for this kind of benchmark, not extra caution: a system that looks strong on data it could have trained on or pattern-matched tells you less than a system that holds up against households it has never seen.

Key takeaways

• A usable eval corpus needs an answer key with a real source of truth — either careful hand-labeling with human review, or ground truth emitted by the same deterministic process that created the scenario.
• The adapter contract (ingest once, then answer blind) is what keeps a memory evaluation honest — ground truth must never reach the system under test, enforced in code rather than assumed.
• Score two axes separately: retrieval (precision/recall@k) and answer correctness (typed exact match). Collapsing them into one number hides which kind of failure you're looking at.
• A raw score means little without a baseline. Compare against a verbatim-storage reference to see what your system's compression actually cost.
• Reproducibility rules — no leaked ground truth, matched retrieval budgets, disclosed configuration, held-out reruns — are what separate a defensible benchmark from a vendor's self-reported number.

FAQ