Measuring Memory Compression Loss: A Practical Method

Step 1: Establish the ceiling with a verbatim baseline

The number that makes a compression-loss measurement meaningful is the ceiling it's measured against. Run a memory system that does no compression at all — stores every record close to verbatim, retrieves by simple lexical overlap, answers by pulling the matching field directly out of stored text — over your corpus, at the retrieval budget (k) you intend to test everything else at. This isn't a strawman configuration; a reference implementation built exactly this way (no extraction, no summarization, no graph — just verbatim storage and keyword-overlap retrieval) requires zero network calls and zero API keys, and its whole purpose is to establish the ceiling everything else gets compared against.

On a corpus in the hundreds-to-low-thousands-of-records range, this baseline should score close to the practical maximum — DecisionSynth Bench's own verbatim baseline scores 0.950 overall exact-match on its 591-episode dev set and 0.980 on its held-out set. If your baseline doesn't land somewhere near that range on a comparably sized corpus, something is wrong with the baseline setup itself, not with any system you haven't tested yet.

Step 2: Run the system under test over the identical corpus and budget

Run your actual system — whatever memory pipeline you're evaluating — over the same corpus and the same retrieval budget k used for the baseline. Matching k matters more than it might seem: a larger retrieval budget structurally favors recall for any system, so comparing a baseline run at k=5 against a system run at k=10 doesn't isolate compression loss, it conflates it with a retrieval-budget difference.

Step 3: Diff per task type, not just the overall score

A single blended score hides which kind of structure is actually being lost. Compute the exact-match gap separately for each question type your corpus tests — direct recall, rationale lookup, rule attribution, precedent search, temporal ordering, or whatever task taxonomy applies to your domain — rather than reporting one aggregate delta.

const baseline = await score(runBaseline(corpus, tasks), tasks);
const system    = await score(runSystem(corpus, tasks), tasks);

for (const taskType of TASK_TYPES) {
  const gap = baseline.by_type[taskType].exact_match_rate
            - system.by_type[taskType].exact_match_rate;
  console.log(taskType, gap.toFixed(3));
}
// A gap concentrated on rule_attribution specifically means the
// extraction step is dropping citations, not general recall.
// A gap spread evenly across every type points at a different
// problem — likely retrieval, checked in step 4.

Step 4: Attribute the loss to retrieval or to extraction

A lower exact-match score has two structurally different possible causes, and the fix is different depending on which one applies. If retrieval recall (R@k) is also low relative to the baseline on the same task type, the system isn't finding the relevant evidence at all — that's a retrieval problem, and the fix is in how the system searches or indexes, not in what it extracted from what it found. If retrieval recall is close to the baseline but exact-match is still low, the system found the right evidence and still answered incorrectly — that's an extraction or reasoning problem, meaning whatever got stored from that evidence didn't preserve what the question needed, even though the evidence itself was retrievable.

This is the step that turns 'the system lost 40 points on rationale lookup' into an actionable finding: 'the system loses 40 points on rationale lookup because recall stays high but the extracted representation doesn't preserve override reasons' points an engineering team at the extraction prompt. 'The system loses 40 points on rationale lookup because recall itself is low' points them at the retrieval index instead. Reporting the blended score alone erases that distinction entirely.

Step 5: Reading the table the way a scoreboard is meant to be read

Once the baseline, the system, the per-task-type gaps, and the retrieval-vs-extraction attribution are all in hand, reading the resulting table is mostly a matter of resisting two temptations: treating the baseline's high score as an embarrassment for the system under test rather than the reference point it's meant to be, and averaging away exactly the per-task-type detail that made the measurement useful in the first place. A system that trails the baseline by a wide margin overall but tracks it closely on the task types that matter most for your actual use case may be perfectly fit for purpose — the baseline exists to show what compression costs, not to set a bar every production system is expected to clear.

Key takeaways

• Compression loss is measured, not assumed — run a verbatim, no-compression baseline over your own corpus at your own retrieval budget as the reference point.
• Match the retrieval budget (k) between the baseline and the system under test — a larger budget structurally favors recall regardless of pipeline quality, which would conflate two different variables.
• Diff per task type, not just overall — a gap concentrated in one question type is a specific, fixable finding; a gap spread evenly across every type usually points somewhere else entirely.
• Attribute each gap to retrieval or extraction by checking recall alongside exact-match — low recall means the system isn't finding the evidence; high recall with low exact-match means it found the evidence and still lost the structure the question needed.
• The baseline's high score is the ceiling reference, not a bar every production system needs to clear — the useful output of this method is the diagnosis, not a verdict that any gap is a failure.

FAQ