Temporal knowledge graphs win exactly one thing (and it matters)

On DecisionSynth Bench's held-out scoreboard, Zep/Graphiti's best configuration scores 0.215 overall exact-match — the lowest of the three vendor systems tested, well behind Letta's 0.917–0.922 and Mem0's best row of 0.465. Read one column further and the ranking inverts: on temporal-ordering questions specifically — what preceded a given decision — Graphiti (Gemini-creator) scores 0.886, the best of any LLM-backed system in the scoreboard, ahead of Letta's 0.760 on the same column. Only the deterministic, non-LLM baseline scores higher, at 0.982.

An overall-EM leaderboard would rank Graphiti last. A leaderboard filtered to the one task its architecture was purpose-built for ranks it first among every system that involves a language model at all. Both rankings are accurate. Neither is complete on its own.

Why the architecture wins exactly this task

A temporal knowledge graph represents facts as edges between entities, and — specifically in Graphiti's design — each edge carries an explicit validity window: when it became true, when (if ever) it was superseded. That's a structural match for a question like "what decisions preceded this one within the past two years," because the graph doesn't need to reconstruct chronology from unstructured text at query time — the ordering is already encoded in the data model itself. A passage-retrieval or fact-extraction system has to infer order from timestamps embedded in whatever text it retrieved; a temporal graph has order as a first-class, queryable property.

Notice the spread within Graphiti's own row: temporal EM ranges from 0.389 to 0.886 depending on the creating model (a gap covered in detail in a companion piece on extraction-volume variance) — and even at its worst, that range still overlaps with or exceeds the other two systems' best temporal scores. The architecture's advantage on this specific task holds even when its own configuration quality varies substantially.

The same architecture, the same run, losing badly at something else

Graphiti's edge-level retrieval unit is also exactly why it loses so badly elsewhere. One decision episode decomposes into roughly 25 individual edge facts under this pipeline. At the benchmark's matched 5-item retrieval budget, a query for "what was the outcome of episode X" competes against every other edge extracted from that episode and from distractor episodes for a slot in five returned items — so even when the graph provably contains the fact a task needs, the fixed budget frequently doesn't surface it. The held-out exact-identifier recall for the strongest Graphiti configuration is 0.110, against 0.994 for the verbatim baseline on the same underlying task shape.

The pattern repeats on the public dev set

The same shape — worst overall score, best temporal score — isn't unique to the held-out run. Graphiti's single-creator dev-set row (591 episodes, publicly reproducible from a clean clone) scores 0.128 overall EM against a 0.950 baseline, and 0.786 temporal-ordering EM against the baseline's 0.907 on the same column — the closest any of Graphiti's numbers come to matching the baseline on any task type, dev or held-out. The same row's precedent-search EM is 0.000 and rationale-lookup EM is 0.009, both near-total losses on tasks that need a complete record rather than an ordering signal.

That 0.121-point gap on temporal ordering is by far the narrowest of any column in Graphiti's dev-set row — on every other task type, the gap to baseline exceeds 0.9. Two independently generated corpora, over disjoint households, producing the same lopsided profile is what separates "this architecture has one narrow strength" from "this run happened to favor one column."

Architecture fit, not architecture ranking

The generalizable claim here isn't "temporal graphs are good" or "temporal graphs are bad" — an overall ranking answers neither question honestly, because the answer depends entirely on which task the deployment actually needs. The generalizable claim is narrower and more useful: retrieval-unit granularity is a first-order design variable, and a system's fitness for a specific task follows from that choice directly, independent of how the same system performs on a different task with different structural needs.

For a team evaluating agent memory for a use case where "what happened, in what order" is a real, recurring question — an audit trail, a sequence-of-events reconstruction, a compliance narrative — a temporal knowledge graph's edge-level, time-stamped representation is doing something no passage- or fact-level system in this scoreboard does as well. For a use case where "what exactly was the outcome of this one specific decision" dominates, the same architecture's retrieval-budget crowding is a real, measured cost that an overall-EM number would bury under everything else the system does poorly.

What this means for reading any memory-system leaderboard

A single overall number necessarily discards the information that makes this finding legible at all. Any team comparing memory-system vendors on the strength of one leaderboard score is making the same mistake an overall-EM ranking of this scoreboard would make: concluding Graphiti is simply worse, when the accurate statement is that it's better at exactly one thing that a general-purpose ranking doesn't weight, and worse at several things that ranking does weight heavily. The fix isn't a smarter aggregate formula — it's reading the per-task-type columns for the task your deployment actually has.