The same model extracts 18 facts or 3, depending on the pipeline

Run the same 543 held-out decision episodes through the same temporal-knowledge-graph pipeline, and swap only the model doing the extraction: Claude-haiku pulls out roughly 2.8 edge facts per episode. Gemini-flash pulls out roughly 18. Nothing else about the task changed — same episodes, same graph-construction prompt, same downstream schema. The six-fold difference is entirely a property of which model sat inside the extraction step.

That gap comes from DecisionSynth Bench's cross-provider scoreboard — every vendor pipeline runs twice, once with gemini-2.5-flash and once with claude-haiku-4-5 as the creating model, specifically to isolate exactly this kind of effect. Under Zep/Graphiti's temporal-knowledge-graph pipeline, the disparity is the largest of any vendor in the scoreboard, and it's the clearest illustration of a finding worth taking seriously: extraction volume is not a fixed property of a model's capability. It's a property of the pipeline the model is embedded in.

The downstream cost of under-extraction

A sparser graph isn't just a smaller number — it's specific, measurable retrieval loss. On the held-out set, the Claude-creator Graphiti configuration posts R@5 of 0.425 and temporal-ordering exact-match of 0.389. The Gemini-creator configuration, extracting roughly 6x more edge facts from the same episodes, posts R@5 of 0.737 and temporal EM of 0.886 — the best temporal-ordering score of any LLM-backed system in the entire scoreboard, ahead of every extraction-based system and every archival system except the deterministic no-LLM baseline.

A temporal knowledge graph's entire value proposition is that it represents facts as time-stamped edges, so a question like "what happened before this decision" can be answered by walking the graph rather than reconstructing order from unstructured text. That value proposition depends on the graph actually containing enough edges to walk. A pipeline that extracts 2.8 facts per episode has, in a meaningful sense, built a much thinner graph than the same architecture is capable of — not because the architecture is wrong, but because the extraction step upstream of it under-delivered.

Why a pipeline can suppress extraction volume

Three pipeline-level factors plausibly explain a gap this size, and they're worth separating because they call for different fixes:

• Extraction prompt design. What the prompt asks the model to pull out — and how explicitly it enumerates the fields worth extracting — shapes yield independent of the model's raw capability to identify those fields if asked directly.
• Context window given to the extraction step. A pipeline that shows the model less of the source episode per extraction call will structurally extract less, regardless of which model is doing the extracting.
• What counts as "salient" in that pipeline's design. A temporal-graph pipeline optimized for precision over recall will under-extract by design; the same model asked to maximize recall would likely extract more, at some cost to graph quality per fact.

DecisionSynth Bench's harness doesn't isolate which of these three specifically drove the 2.8-vs-18 gap for this pipeline — that would require an ablation study beyond what the scoreboard's maintainer runs test. What the scoreboard does establish, cleanly, is that the gap exists and that it's attributable to the pipeline-model interaction rather than to the corpus or the task.

The same pattern shows up on a second, independent corpus

The 2.8-vs-18 figure comes from the held-out set, but the underlying instability isn't a one-corpus artifact. On DecisionSynth Bench's separate 591-episode public dev set — a different sample of episodes over different households — the single-creator (Gemini) Graphiti configuration posts a temporal-ordering EM of 0.786 and an overall EM of 0.128. The held-out Gemini-creator row posts temporal EM 0.886 and overall EM 0.215 — broadly consistent with the dev-set numbers, which is what you'd want to see from a pipeline behaving predictably. The Claude-creator configuration wasn't run on the dev set in the same breakdown, but the magnitude of its held-out shortfall relative to Gemini (temporal EM 0.389 vs. 0.886, a gap of exactly the kind the extraction-volume difference would predict) is consistent with a pipeline-level effect rather than a held-out-set-specific fluke.

The Gemini-creator configuration's numbers move together, consistently, across two independently sampled corpora — which is the pattern a stable pipeline should produce, and the pattern this specific configuration does produce. It's the Claude-creator configuration, extracting roughly six times fewer facts per episode, that departs from it on the held-out run.

What this means for evaluating a memory vendor

The practical consequence for anyone evaluating a memory system: asking "how good is Vendor X's extraction" is an underspecified question until you also specify which model is running inside their pipeline. A vendor benchmark that reports a single score per product, with no visibility into which model produced it, is withholding a variable that — on this evidence — can swing the outcome by 6x on extraction volume and roughly 2x on the downstream retrieval and ordering metrics that depend on it.

For teams building or buying a memory pipeline, the actionable version of this finding is: if extraction volume matters to your use case (and for anything requiring cross-episode structure — precedent search, temporal ordering, multi-record aggregation — it does), test your actual creator model against your actual pipeline rather than trusting a competitor benchmark run on a different model. The gap is large enough, and pipeline-specific enough, that a generic "this vendor extracts well" claim doesn't transfer across model choices.