Retrieval Granularity
Retrieval granularity is the size of the unit a memory system retrieves when answering a question — a whole passage, a single extracted fact, or one edge in a knowledge graph — and it's a first-order design choice because a fixed retrieval budget (the top-k items returned) carries a very different amount of any one source record depending on how finely that record was broken up at storage time.
Three retrieval units dominate current systems. Passage-level retrieval returns a chunk of original text more or less as stored — high fidelity per item, but coarse, since a single passage can contain much more (or less) than what's actually relevant to the question. Fact-level retrieval, the unit an extraction pipeline like Mem0's produces, returns discrete statements — precise, but only as complete as what the extraction step chose to pull out. Edge-level retrieval, the unit a temporal knowledge graph like Graphiti produces, returns individual (entity, relationship, entity) triples — the finest-grained of the three, and the unit best suited to representing how a fact changes over time, since each edge carries its own validity window.
Granularity matters because a fixed top-k retrieval budget behaves differently depending on the unit size. At the same k, passage-level retrieval returns k chunks of context; edge-level retrieval returns k individual triples, which for a single complex record can mean the budget is spent on fragments of one source rather than a complete picture. That's a real tradeoff, not a defect of finer granularity — the same fine-grained structure that risks crowding out a complete record at a small budget is exactly what makes a temporal graph able to answer an ordering question (what happened first, second, third) that passage-level retrieval isn't naturally shaped to answer at all.
The practical implication: which granularity performs best is task-dependent, not universal. A question asking for one specific fact rewards fine granularity precisely on target; a question asking for a complete decision's full context rewards coarser granularity that keeps a record together; a question asking about order across multiple decisions rewards the graph structure that fact- or passage-level retrieval doesn't represent at all.
| Passage | Fact | Edge | |
|---|---|---|---|
| Typical source | Extraction pipeline output | Fact-extraction system (e.g. Mem0) | Temporal knowledge graph (e.g. Graphiti) |
| Best suited to | Complete-context questions | Single discrete-fact questions | Ordering / temporal-change questions |
| Risk at fixed k | Coarse — may include irrelevant text | Moderate | Budget can fragment across one record |
Benchmarking a memory system's retrieval quality only makes sense at a matched budget across systems with different native granularities — comparing 5 passages against 5 individual graph edges isn't an apples-to-apples comparison of how much source material each system actually surfaced. Reporting per-task-type scores (not just an aggregate) is what makes a granularity mismatch visible rather than averaged away.
Common pitfalls
- Assuming finer granularity is strictly better because it's more precise — it also means a fixed retrieval budget covers less of any single complex record, which can hurt tasks that need a complete picture rather than one isolated fact.
- Comparing retrieval scores across systems with different native units at the same raw k without accounting for what k actually represents in each system.