Decision-Recall Eval Pack
Every agent-memory vendor claims their system remembers what was decided for a client and why. Almost none of them can prove it, because proving it requires a decision history where the ground truth is actually known — every decision, every override, every outcome — and real client histories are unlabelable, private, and legally radioactive. The Decision-Recall Eval Pack solves this by construction: 150 retrieval-mode tasks over synthetic advisor-decision episodes whose answer keys were emitted by the same deterministic generator that produced the episodes. The question “did the client ever deviate from the recommendation, and when?” has one correct answer, and we know it because we generated the deviation.
Preview a task
Decision-recall tasks are never published — none appear in the free split, and that scarcity is deliberate: a memory benchmark a model may have seen is not a benchmark. Every task follows this contract:
"prompt": a question answerable only from the client's recorded decision history "recall_context": episode ids to load into your memory system — in_context or retrieval delivery "answer_key": typed expectations + text assertions, known by construction "license": training_use_permitted: false
What ships in the ZIP
Plus the AI Eval Sets methodology PDF, your license certificate, and receipt as separate account downloads. Data ZIP downloads are limited to 3 per order.
Why these tasks discriminate
In-context memory saturated at 1.00 for every system in the pilot of record — with the episodes in the prompt, frontier models don't forget. Retrieval mode is where systems separate: these tasks make your memory pipeline find the right episodes first, then answer from them, scored against keys emitted by the episode generator itself.
Pre-registered protocol, k=3, published substitution log — see the scoreboard.
Run your evaluation
- 1Present each blinded task to your system, three attempts each.
- 2Collect answers as JSON —
{task_id, attempt, text}. - 3Run
npx tsx score.ts tasks.jsonl answers.json --k 3. - 4Report pass@1, pass^3, and SFR — with the denominator and vintage.
Seed your agent's memory with DecisionSynth packs; grade it with this one. Same episodes, opposite sides of the eval.
Why this Data Set exists
Memory is the least-measured layer of the AI-assistant stack. Teams evaluate their models, evaluate their retrieval, and then take the memory product's behavior on faith — because building a memory benchmark honestly means authoring decision histories with known ground truth, which is weeks of expert labeling that goes stale the moment the format changes. So procurement decisions get made on demos, and demos are curated.
The failure modes memory systems actually exhibit are specific and testable: conflating two similar decisions from different months, attributing an override to the wrong party, reporting the recommendation instead of what was actually done, and — most dangerous — confidently answering from general knowledge when the recorded history contains nothing on the topic. None of these show up in a demo. All of them show up under a task set that asks pointed questions against a history where every fact is known.
This pack is that task set. Each task supplies a client's decision-episode history and asks a question a working advisor would ask their assistant: what was decided about the 529 in March, why was the rebalance overridden, how did the Roth conversion turn out, how many times did the client go against the recommendation. Keys are structured and graded mechanically. Because the episodes are synthetic, you can hand the corpus to any vendor without a privacy review — and because the held-out pool is drawn from a disjoint episode set, a vendor who has seen this pack still can't game your follow-up.
Use Cases
Who uses this Data Set
Publishes scores on a corpus their system has never seen instead of a curated demo — and runs the pack in CI so retrieval changes that break decision recall fail before release, not in a customer's pilot.
Puts competing memory vendors on identical tasks with identical keys. The pack's episodes load into any memory system's ingestion format, and mechanical grading makes the vendor comparison a table, not a judgment call.
Scores the same tasks in both delivery modes — episodes embedded in-context versus served through the retrieval stack — to isolate whether recall failures come from the model or from retrieval.
Regression-tests the assistant's client-history features after every model or prompt change, with special attention to the tasks where the correct answer is “the history doesn't say” — the hallucination-under-pressure cases.
Studies decision-relevant memory with a corpus where ground truth is generative rather than annotated — no inter-labeler disagreement, no label noise, and a regeneration path for scale (licensed research use).
What's inside
150 decision-recall tasks in JSONL, all commercial — decision-recall is never in any public split, so nothing in this pack has ever been published. Each task carries a system prompt that scopes the assistant to the recorded history, a user question, and a recall_context naming the household and the exact episode IDs the answer depends on. The referenced DecisionSynth episodes ship in the pack, so you control delivery: embed them in-context to test the bare model, or load them into your memory or RAG product and let your stack do the retrieval — the tasks and keys are identical either way, which is what makes bare-model versus memory-product comparisons clean.
Task templates cover the recall behaviors that matter in production: what was decided and when, the reasoning and cited policy behind a recommendation, override attribution and stated reasons, outcome recall, cross-episode counting (how many times did the client deviate?), and negative probes — questions like “was any deviation ever escalated beyond supervisor review?” where the correct answer is that it never happened, which is exactly where systems invent history.
Answer keys are structured (expected_structured plus required_text assertions with match-any variants) and graded by the included mechanical scorer at k=3 runs per task. Episodes and tasks are generated deterministically — version-stamped, byte-identical on regeneration — and are disjoint by construction from the free DecisionSynth Bench dev set, so public-set contamination is impossible.
Technical Highlights
Sample Schema Fields
{
"task_id": <value>,
"recall_context.delivery": <value>,
"recall_context.episode_ids[]": <value>,
"prompt.user": <value>,
"answer_key.expected_structured": <value>,
"answer_key.required_text[]": <value>,
"grading.method": <value>,
"provenance.seed": <value>
}Sample queries
Load the pack's episodes into the memory system under test, run the task prompts against it, and grade the transcript file mechanically.
npx tsx score.ts tasks.jsonl answers.json --k 3 # per-task: match expected_structured + required_text assertions
Running the same tasks in-context and through retrieval isolates where recall breaks: a task that passes in-context but fails through the stack is a retrieval bug, not a model bug.
taskIds.map(id => ({
id,
inContext: inContextRun[id].pass,
retrieval: retrievalRun[id].pass
})).filter(r => r.inContext && !r.retrieval)The tasks where the correct answer is “that never happened” — the cases where memory systems hallucinate most confidently. Template IDs make them addressable directly.
tasks.filter(t =>
["dr-escalation-search", "dr-no-override-probe"]
.includes(t.provenance.template_id)
)Each task names its household by archetype and seed, so you can evaluate recall depth on a single long history end-to-end.
tasks.filter(t => t.recall_context.household.archetype_id === "A-01" && t.recall_context.household.seed === 1 )
Methodology
The episodes come from the DecisionSynth generator: deterministic, seeded advisor-decision records — trigger, options, recommendation, override, outcome, cited rule keys — over synthetic households spanning the archetype library. The recall tasks are emitted in the same generation run, so every key is derived from the episode data itself rather than labeled after the fact; there is no annotation step to introduce noise. Grading is mechanical (structured matching plus text assertions, k=3 runs, no LLM judge). The corpus is version-stamped and byte-identical on regeneration, and it is disjoint from both the free DecisionSynth Bench dev set and the private held-out pool, which is reserved for scored comparisons that must stay uncontaminated. Zero PII throughout: every household, decision, and outcome is synthetic.
Frequently asked questions
How can the answer keys be trustworthy without human labeling?+
Because they were never labeled — they were generated. The episode generator decides the override in month 91, so the key to “when did the client deviate?” is month 91 by construction. Human labeling would only add a lossy transcription step to facts the generator already knows. This is the same correct-by-construction doctrine as the rest of the WealthSynth line: ground truth is an output of generation, not an annotation applied afterward.
What does “dual delivery” mean in practice?+
The tasks are retrieval-mode: each names the episode IDs its answer depends on, and the episodes ship in the pack. You choose how they reach the model — embed them directly in the prompt (in-context mode, testing the bare model's reading of the history) or load them into your memory or retrieval product and let it supply them (testing the stack). Same tasks, same keys, so the two scores are directly comparable and their difference is attributable to your retrieval layer.
Why isn't there a public split for decision-recall?+
Deliberately. The free DecisionSynth Bench dev set already gives the public a way to see episode structure and task style. Keeping every commercial recall task unpublished means a vendor can't have incidentally trained on any of it, and the private held-out pool — drawn from a disjoint episode set — stays available for scored comparisons even against systems that have purchased this pack.
Does this test whether the model refuses to invent history?+
Yes — the negative-probe templates ask about events that never occurred in the recorded history (an escalation that never happened, an override that was never made), and the key requires the assistant to say so plainly. These catch the most dangerous production failure: a memory system that answers from the model's general knowledge when the history is silent, which reads as perfect recall right up until it fabricates a client interaction that never happened.
Why is this pack not vintaged like the TY2026 packs?+
Because nothing in it rolls with the tax year. Recall tasks test memory of recorded decisions, not current-year figures, so the pack doesn't expire in January. Where episodes cite regulatory figures, the citations are to versioned fact keys that were correct as of the episode's date — which is itself part of what a good memory system should preserve.
How does this relate to the DecisionSynth Data Sets?+
Opposite sides of the same evaluation. The DecisionSynth packs are seed and training-adjacent data: decision histories you load into a memory product. This pack is the held-back exam: tasks and keys over episodes your system hasn't seen. Teams building on DecisionSynth data buy this pack to grade what they built; the episode sets are disjoint, so the exam stays fair.
Can we train on these tasks or episodes?+
No — evaluation use only (training_use_permitted: false on every task). If you want decision episodes to train or seed a memory system, that's exactly what the DecisionSynth Data Sets are for; keeping the two corpora separate is what makes your scores on this one meaningful.
Related Wealth Data Sets
The complete commercial decision corpus: 1,979 deterministic advisor-decision episodes with 4,913 QA evaluation tasks and answer keys over 1,752 fresh synthetic households spanning all 71 client archetypes. Every episode records trigger, options, recommendation, override, outcome, and cited IRS/Treasury rule keys — ground truth known by construction, version-stamped in _meta, byte-identical on regeneration. Disjoint by construction from the free DecisionSynth Bench dev set and the private held-out set.
Every sellable task in the AI Eval Sets line, all vintages to date: 401 tasks across rule-grounding (156), threshold-cliff (95, incl. the 36-task adversarial flip family), and decision-recall (150), with answer keys, forbidden-figure tables, boundary tables, and the mechanical scorer. Held-out-split exclusions are documented in the manifest: the 70 held-out tasks — including all 40 pilot tasks — are never sold, so a future scored comparison against your deployed system stays meaningful.
156 rule-grounding tasks over IRS/SSA/CMS 2026 figures — each with the correct value cited to its source document (Notice 2025-67, Rev. Proc. 2025-32, CMS 2026) and the enumerated wrong-but-plausible answers: the 2025 value, the 2024 value, superseded and derived-not-published variants, each with a reason code. Ships with answer keys, the primary-source-verified prior-year figure tables, and a mechanical scorer. Score any model's Stale Figure Rate in an afternoon; re-buy the vintage when the figures roll.
Purchases are for internal use only. Redistribution, resale, posting online, or sharing outside your organization is prohibited under the WealthSchema Data License.
Digital product, delivered electronically. All sales are final — no returns, refunds, or exchanges. Defective or incomplete deliveries are corrected free of charge. Data ZIP downloads are limited to 3 per order; account access, your license certificate, and receipt remain available.
Evaluation product: no training use permitted. Corrections to the purchased tax-year vintage within 180 days are provided free.
View data license →