Agent-Memory Vendor Evaluation Checklist
A memory-vendor benchmark number is only as trustworthy as the process behind it, and most published comparisons don't show that process — just the score. This checklist enumerates the five structural questions worth asking any agent-memory vendor before treating their number as comparable to anyone else's: how the ground truth was produced, whether a genuine held-out protocol exists, whether every system was compared at the same retrieval budget, which model actually created the answers, and whether any of it can be reproduced independently. A vendor with good answers to all five isn't necessarily the best-performing system — but a vendor who can't answer them is asking for trust their own number hasn't earned.
Label provenance
- How was the ground truth produced?
Ask directly: hand-labeled by a human after the fact, or emitted by a deterministic process at the moment the scenario was created. Both are legitimate — what matters is that the vendor can answer specifically, rather than deflecting to 'our eval team reviewed it.'
- If hand-labeled, was there independent review?
A single annotator's judgment call, unreviewed, carries more disagreement risk than a process where a second person (or a documented review step) checked the label. Ask what the review step actually was, not just that one existed.
- Is there circularity between the question-generation model and the scoring model?
If the same model family that seeded the questions is also being scored as a top performer, that's worth scrutinizing — ask what independent check exists to rule out the model simply being well-matched to its own question style.
Held-out protocol
- Does a genuine private held-out set exist?
A benchmark with only a public, answer-visible split tells you how a system performs on data it could have been tuned against. Ask whether a separate, unpublished split exists and whether the published score comes from that split specifically.
- Is disjointness from the public set asserted and checkable, not just claimed?
Ask how overlap between the held-out and public sets is prevented or verified — a build-time assertion is a stronger claim than an unverified statement that the sets 'don't overlap.'
e.g. "households/entities asserted disjoint at build time" vs. an unsupported claim - Are held-out answers ever published?
If a vendor's held-out answer key is publicly available anywhere, it's not functioning as a held-out set anymore — the entire point is that a system under test cannot have seen the correct answers in advance.
Retrieval budget parity
- Was every system in the comparison run at the same retrieval budget (k)?
A larger retrieval budget structurally favors recall for any system, regardless of pipeline quality. If a vendor's own system was tested at a larger k than the systems it's compared against, the comparison isn't measuring pipeline quality alone.
- Is the retrieval budget disclosed at all?
A published score with no stated k isn't reproducible or independently checkable — ask for the specific number, not just "a reasonable retrieval window."
- Is the native retrieval unit disclosed, so a raw k can be interpreted correctly?
The same k means very different amounts of retrieved content depending on whether the native unit is a whole passage, a single extracted fact, or one graph edge — a single decision record can shred into dozens of edges. Ask what unit k is counted in before comparing raw numbers across systems with different architectures.
Creator-model disclosure
- Which LLM powers extraction and answer generation for each vendor row?
Most memory pipelines embed an LLM for extraction, answer generation, or both. A score with no disclosed creator model is really reporting pipeline-plus-unknown-model performance, which isn't reproducible or comparable to a row where the model is disclosed.
- Was more than one creator model tested for the same pipeline?
The same underlying architecture can swing substantially depending on which model sits inside it — in one documented case, the same architecture extracted roughly 18 facts per record under one model and roughly 2.8 under another. A single-model row hides how much of the score is architecture versus model choice.
- Were embeddings held constant across the vendors being compared?
If retrieval embeddings differ between vendor rows in the same comparison, that's another uncontrolled variable stacked on top of the pipeline and creator-model differences — ask whether embeddings were standardized across the comparison.
Reproducibility
- Can a baseline or reference number be reproduced from a clean clone?
A defensible benchmark's simplest reference point — typically a no-compression, verbatim-storage baseline — should be reproducible with minimal setup, ideally with zero API keys required, since it exists specifically to give anyone a checkable ceiling reference.
- Is the exact per-vendor configuration published?
Model names, index settings, and dependency versions needed to reproduce a specific vendor row should ship alongside the score. A score with no configuration attached isn't reproducible, and a number that isn't reproducible isn't really a score.
- Is there a documented adapter or integration contract a new system could implement to get a comparable number?
A benchmark built around a narrow, published ingest/answer contract signals the comparison was designed for fairness across systems, not built ad hoc around whichever vendors happened to be tested first.
Key takeaways
- A vendor benchmark number is only as trustworthy as its ground-truth provenance — ask directly whether answers were hand-labeled and reviewed, or generated deterministically at creation time.
- A held-out protocol only defends against overfitting if it's genuinely private, asserted disjoint from public data, and never published — a claimed-but-unverified split isn't the same thing.
- Retrieval budget and native retrieval unit both have to be disclosed and matched across systems, or the comparison is measuring budget differences as much as pipeline quality.
- Creator-model disclosure matters because the same architecture can swing substantially with a different underlying model — a single-model row hides how much of a score is architecture versus model choice.
- Reproducibility — a checkable baseline, published configuration, a documented adapter contract — is what separates a defensible benchmark from a vendor's self-reported number.