"Our agent has memory" is one of the easiest things to claim and one of the hardest things to test. Retrieving a note from a vector store is not the same as remembering why a client overrode a recommendation last spring, in the right order, attributed to the right rule. In consumer chat, a memory slip is an annoyance. In wealth, tax, or advice, it is the difference between an agent that can be trusted with a decision and one that cannot.
The trouble is that most teams evaluate memory by looking at outputs and deciding whether they seem right. Eyeballing does not scale, does not regress, and does not survive a disagreement about what "right" means. To evaluate memory honestly you need the one thing that is hardest to get in this domain: ground truth you did not write after the fact.
Why financial decision memory is a distinct problem
An agent operating over a client's history has to do more than recall facts. Real tasks look like:
- Decision recall: what was decided, and when.
- Rationale lookup: why it was decided, including the reason for an override.
- Precedent search: has a similar situation come up before, and what happened.
- Temporal ordering: which decision came first, and what changed between them.
- Rule attribution: which regulation or figure governed the choice.
Each of these can fail independently. An agent can recall the decision but not the reason, or the reason but not the rule, or all three but in the wrong order. A single "does it seem to remember" score hides exactly the failures that matter.
The ground-truth problem
To score those tasks you need to know the correct answer. There are two ways to get one, and only one of them is trustworthy.
You can label examples after the fact. A human reads a transcript and writes down what the answer "should" be. This is slow, it is expensive, and it imports the labeler's mistakes and assumptions into your benchmark. Worse, the thing you are testing (a system that reasons over decisions) is being graded by another fallible process that reasoned over the same decisions.
Or you can generate the data so that the answer is known by construction. If a deterministic generator produces the decision and emits the answer key at the same time, the key is correct because the generator made it so, not because someone judged it later. Nothing is hand-labeled, so nothing is mislabeled.
What "known by construction" means
The stronger approach builds each decision episode as a deterministic projection of a synthetic household's trajectory through a policy engine. No language model writes a field. Every value derives from the household, its trajectory, its life-event schedule, or a cited rule. A value that cannot be traced back to one of those is treated as a bug, not an answer. Same seed, same output, every time.
Because the episode is generated rather than observed, the QA tasks that come with it can carry answer keys, evidence identifiers, and deterministic distractors, emitted by the same generator. You are not grading against a rubric you cannot inspect. You are grading against a key the data itself produced.
episode: {
"trigger": "promotion_or_raise",
"followed_policy": false,
"override_reason": "bracket_timing_judgment",
"semantic_refs": ["retirement.401k.elective_deferral.under_50"]
}
qa: {
"task_type": "rationale_lookup",
"question": "Why was policy overridden?",
"answer_key": "bracket_timing_judgment"
}
The episode records what happened and which cited figure governed it. The QA task asks a question a memory system should be able to answer, and carries the answer the generator already knew. You grade against the key, not a vibe.
Two jobs, one dataset
Ground-truth decision data does two things for a team building agents:
- Grade what you have. Point your memory pipeline at episodes whose answers were known first, and score recall, rationale lookup, precedent, ordering, and rule attribution as separate dimensions instead of one fuzzy impression.
- Seed what you are building. Cold-start a new agent with a realistic decision history on day one (who decided what, why, what policy said, how it turned out) without touching a production CRM or any real client's data.
How to run the evaluation
- Fix your task set: pick the memory dimensions that matter for your product (most teams start with recall, rationale lookup, and rule attribution).
- Hold out a set the agent never sees during development, so you are measuring memory, not overfitting.
- Score per dimension against the answer keys, and track each dimension over time so a regression in "why" does not hide behind a good "what."
- Compare honestly. A public benchmark you can inspect is worth more than a private number you cannot defend.
WealthSchema's DecisionSynth is built for exactly this: 1,979 deterministic advisor-decision episodes and 4,913 QA tasks with answer keys emitted by the generator, plus a free public benchmark you can inspect before you buy anything. If you are evaluating agent memory in finance, the fastest way to judge it is to look at the benchmark and the answer keys yourself.
Bottom line
You cannot grade what you cannot check. Evaluating an agent's financial memory needs decision data with answers that are correct by construction, scored per dimension, on a held-out set. Eyeballing is not a benchmark.