Why AI Agents Forget Decisions — and How to Measure Exactly What's Lost

What "forgetting" actually means in a memory pipeline

There are two different things an agent can fail to do, and conflating them hides the useful diagnosis. True deletion is when the source record itself is gone — a retention policy purged it, a database row was dropped. That's rare and usually intentional. Far more common is retrieval failure caused by lossy indexing: the memory system read the source record once, extracted what its pipeline considered salient, and discarded the rest — so when a question later needs something outside that extracted subset, there's nothing left to retrieve, even though the original record may still exist upstream.

This is what 'the agent forgot' usually means in practice: not that the fact was deleted, but that the memory layer's summary of it never captured the specific detail — a rationale, an identifier, a citation — that the question turns out to need. The failure happened at write time, when the pipeline decided what was worth keeping, not at read time when the question was asked.

The verbatim baseline as the ceiling reference

To measure a loss, you need something to measure it against. The reference point is a memory system that does no compression at all: store every record verbatim, retrieve by simple lexical overlap, answer by pulling the matching field directly out of the stored text. No extraction, no summarization, no graph — just grep. That baseline isn't a strawman built to make real systems look bad; it's the closest thing to a hard ceiling this kind of evaluation has, precisely because it discards nothing.

On DecisionSynth Bench's public dev set — 591 decision episodes, 1,487 QA tasks spanning direct recall, rationale lookup, precedent search, temporal ordering, and rule attribution — the committed verbatim baseline scores 0.950 overall exact-match, with 0.972 recall@5 and 0.292 precision@5 (precision reads low only because it's mathematically capped at 0.2 for any task with a single relevant piece of evidence — the committed scoreboard notes this explicitly, and it's worth checking before reading a precision number as a weakness). A near-0.95 ceiling on a corpus this size confirms the ceiling is real and reachable, not a theoretical maximum nobody actually hits. Any production system's gap below that number, on the same corpus and the same retrieval budget, is a direct measurement of what its compression cost.

Reading the gap: different task types expose different kinds of loss

A single aggregate score hides which structure a system's pipeline is actually dropping, because different question types stress different parts of what got extracted. Direct recall (what did household X decide) mostly needs the outcome field to have survived extraction intact. Rationale lookup (why was the policy overridden) needs the reasoning, not just the result, to have been captured — a pipeline can preserve 'the client took option B' while discarding 'because of a documented liquidity concern' if its extraction step only pulled the decision, not the justification behind it. Rule attribution (which regulatory figure governed the choice) needs a specific citation string to have survived — exactly the kind of low-salience, easy-to-summarize-away detail an extraction pipeline optimized for readability tends to drop first. Temporal ordering and precedent search are different again: they need structure across multiple records, not just fidelity within one, so a system can preserve every individual fact perfectly and still fail if it can't aggregate across records at query time.

The diagnostic move is to compare per-task-type scores, not just the overall number. A system that's strong on direct recall and weak on rule attribution has a specific, fixable problem — its extraction step is dropping citation strings — that a single blended score would never surface.

Why the gap is a design property, not a bug to patch

It's tempting to treat a memory system's gap to the verbatim baseline as a defect to eliminate. It isn't, and the reason is structural: verbatim storage doesn't scale, cost-wise or latency-wise, past a fairly small corpus, which is exactly why every production memory system extracts, summarizes, or graphs instead. The tradeoff is real and unavoidable — you cannot keep full-fidelity, instantly queryable storage of everything a system has ever seen AND keep retrieval cheap and fast forever. Compression buys the scale; the gap to the ceiling is the price.

That reframes the right question. It isn't 'how do we get our memory system to score exactly like the verbatim baseline' — a system that did would have given up the scaling properties that made it worth building in the first place. The right question is 'how much of this specific structure — rationale, citations, precedent — can we afford to lose for our use case, and does our current pipeline lose more than that.' A system feeding a casual chatbot can tolerate a wide gap on rule attribution, because nothing downstream needs a citation. A system feeding an advisor agent under compliance review cannot, because 'which rule permitted this' is exactly the question an examiner asks.

Measuring it on your own system

The method doesn't require guessing. Run the verbatim baseline and your actual system over the same corpus, at the same retrieval budget, and diff the per-task-type scores. The delta on each task type tells you, concretely, what your pipeline's compression is costing — not in the abstract, but as a number attached to a specific kind of information (rationale, citation, cross-record structure) your system either kept or didn't.

// Illustrative — the real harness ships as bench/score.ts
const baseline = await score(runBaseline(corpus, tasks), tasks);
const system    = await score(runSystem(corpus, tasks), tasks);

for (const taskType of TASK_TYPES) {
  const gap = baseline.by_type[taskType].exact_match_rate
            - system.by_type[taskType].exact_match_rate;
  // gap > 0 on rule_attribution specifically? your extraction
  // step is dropping citations, not general recall.
}

Key takeaways

• "Forgetting" in a memory pipeline is almost always lossy indexing at write time, not deletion — the source record often still exists, but the compressed index of it doesn't contain what the question needs.
• A verbatim-storage baseline is the ceiling reference: on DecisionSynth Bench's dev set it scores 0.950 exact-match, confirming the ceiling is reachable, not theoretical.
• Different task types expose different kinds of loss — rationale, rule citations, and cross-record structure each fail independently, so a blended score hides which one your pipeline is actually dropping.
• The gap to the baseline is a design tradeoff, not a bug — compression is what lets a memory system scale past raw verbatim storage. The job is measuring the gap and deciding what your use case can tolerate, not eliminating it.
• Measure your own system by running it and the baseline over the same corpus at the same retrieval budget, then diff per task type — the gap on a specific task type points at a specific, fixable extraction problem.

FAQ