The Adapter Pattern for Memory Evaluation: One Contract, Any Vendor

The contract itself

The interface is deliberately small. A memory adapter implements two methods: one to load the full corpus once, and one to answer a single task using whatever the system retrieves from its own memory. Nothing else is required, and nothing else is permitted to leak through it.

interface MemoryAdapter {
  name: string;
  // Called once with the full corpus before any questions.
  ingest(episodes: EpisodeRecord[]): Promise<void> | void;
  // Ground truth is NEVER passed in — the runner strips it first.
  answer(task: Omit<QaTask, "answer_key" | "evidence_ids" | "distractor_ids">):
    Promise<SystemAnswer> | SystemAnswer;
}

interface SystemAnswer {
  task_id: string;
  evidence_ids: string[];               // what the system retrieved, best-first
  answer_key: Record<string, unknown>;  // typed answer, same shape as the real key
}

Why one contract per vendor, not a shared abstraction across vendors

It's tempting to build a shared abstraction layer that talks to every vendor's SDK through common primitives, on the theory that it would reduce duplicated code. That's the wrong instinct here. Each adapter should be a single, self-contained file apart from that vendor's own SDK — an archival-storage adapter, a fact-extraction adapter, and a temporal-graph adapter genuinely do different things internally, and forcing them through a shared internal abstraction just to save a few lines risks quietly biasing the comparison toward whatever the shared abstraction happens to make easy. The contract is narrow specifically so each adapter is free to do whatever is idiomatic for its own vendor internally, as long as it produces the same shape on the way out.

What the runner strips, and why it's enforced in code

The fairness of the whole comparison rests on one property: no ground truth ever reaches an adapter's answer() method, under any circumstance. That has to be enforced by the runner itself, not by the discipline of whoever writes an adapter — it's the kind of rule that's easy to violate by accident (a debugging session at 11pm, a 'just this once' shortcut) if it depends on someone remembering not to pass a little extra context through.

// The runner strips ground truth before answer() ever sees the task.
for (const task of tasks) {
  const answer = await adapter.answer({
    task_id: task.task_id,
    task_type: task.task_type,
    question: task.question,
    // answer_key, evidence_ids, distractor_ids: never included
  });
  answers.push(answer);
}

The fairness rules that make a row defensible

A published comparison is only worth as much as the rules behind it. The rules that keep an adapter-based comparison honest are simple to state and worth enforcing explicitly rather than assuming everyone involved already agrees to them:

Common pitfalls when writing an adapter

A few mistakes show up repeatedly when teams build their first adapter for a vendor's system. Rendering the source corpus inconsistently across vendors is one — if one adapter feeds a vendor cleanly formatted structured records while another feeds a different vendor raw, noisier text, the comparison is measuring formatting sensitivity as much as memory quality; every vendor should ingest the identical representation of the source content. Mismatched retrieval budgets are another — comparing one system retrieving 5 items against another retrieving 25 isn't a fair read of retrieval quality, even if both numbers seem reasonable in isolation for that vendor's typical usage.

A subtler pitfall is under-using a vendor's native evidence provenance. Most vendor systems expose some form of per-memory origin natively — metadata tags, source references, episodic provenance markers — and an adapter's job includes translating that native provenance into the evidence_ids the scorer expects, rather than leaving evidence_ids empty because the mapping wasn't obvious. An adapter that doesn't populate evidence_ids correctly will look like it has a retrieval problem it doesn't actually have — the retrieval happened inside the vendor's system; the adapter just failed to report it.

Key takeaways

• A narrow ingest/answer contract — not a shared internal abstraction — is what lets fundamentally different vendor architectures be compared through one scorer fairly.
• Ground truth must never reach an adapter's answer() method, and that has to be enforced by the runner stripping it before the call, not left to an adapter author's discipline.
• Each adapter should be a single, self-contained file per vendor, free to do whatever's idiomatic internally as long as it returns the same shape on the way out.
• Fairness rules — matched retrieval budgets, identical source-content rendering, disclosed configuration, held-out reruns — have to be explicit and enforced, not assumed to be shared understanding.
• Populating evidence_ids from a vendor's native provenance (metadata, tags, episodic references) is part of writing a correct adapter — skipping it makes a system look like it retrieved nothing when it actually retrieved the right thing.

FAQ