Worksheet: Designing Decision-Shaped Test Cases for Your Advisor AI
"What could the agent forget" is the right question to start from and the wrong place to stop — it's too broad to design a test suite around. This worksheet breaks it into the five concrete question shapes a decision-memory evaluation actually needs: can the agent recall what was decided, explain why, find precedent across past decisions, put decisions in order, and cite the rule that governed a choice. Work through each section for your own domain and walk away with a scenario-by-scenario test plan, each case with a known expected answer to check the agent against.
What you walk away with
~25 min · 7 sections · 8 fields- An enumerated test case list covering all five decision-memory question shapes for your own agent's domain.
- A documented expected answer per case, so each test is checkable rather than a judgment call.
- A coverage index surfacing which question shapes your team designed for and which got skipped.
- A next-step path into evaluation methodology and vendor comparison once the cases are designed.
Agent scope
What kinds of decisions this agent recommends, executes, or explains.
Direct recall test cases
Can the agent correctly state what was decided for a specific client, without the rationale or context — the baseline capability everything else builds on.
The client situation and the decision that was made.
The client situation and the decision that was made.
The client situation and the decision that was made.
Rationale lookup test cases
Can the agent explain why a decision was made — specifically, why a recommendation was overridden, not just what happened.
The recommendation, the override, and the specific reason.
The recommendation, the override, and the specific reason.
The recommendation, the override, and the specific reason.
Precedent search test cases
Can the agent find every other decision sharing a specific characteristic with the current one — and report what happened in each, not just the closest single match.
The shared characteristic across multiple past decisions.
The shared characteristic across multiple past decisions.
Temporal ordering test cases
Can the agent reconstruct which of several decisions for the same client happened first, second, third — an aggregation question, not a single-record lookup.
Multiple decisions for one client across a time window, with a real sequence.
Multiple decisions for one client across a time window, with a real sequence.
Rule attribution test cases
Can the agent cite the specific rule or policy that governed a decision — the failure mode a compliance review would flag first if it's missing.
A decision where a specific rule determined which options were even on the table.
A decision where a specific rule determined which options were even on the table.
A decision where a specific rule determined which options were even on the table.
Coverage roll-up
How evenly your designed cases cover the five question shapes — a low score in one category is a specific, fixable gap, not just a lower overall number.
Even coverage across all five question shapes (recall, rationale, precedent, temporal, attribution) at 20% each. A skipped category caps the index well below 100% regardless of how many cases the other categories have.
Next steps
Once cases are designed, populate them in a real or synthetic test corpus and run them against the agent's actual memory system — checking each answer against the expected value, not a plausibility judgment. Cases with status below 'verified in CI' shouldn't be treated as covered yet.
Key takeaways
- "Test the agent's memory" only becomes a test plan once it's broken into concrete question shapes — recall, rationale, precedent, ordering, attribution each fail independently and need their own designed cases.
- Rationale and attribution cases are the ones teams design fewest of by default, and they're the ones a compliance review is most likely to ask about first.
- Precedent and temporal cases require multiple related decisions in the scenario, not a single record — they test aggregation, a different capability from single-record recall.
- The status column is a release gate, not a progress bar — a case that's only 'designed' hasn't verified anything about the agent yet.
- Uneven coverage across the five shapes is a specific, visible gap once tracked — a team that only tested recall has tested one-fifth of what a decision-memory system actually needs to get right.
FAQ
Do we need cases in all five categories, or can we prioritize?
Prioritize based on what your agent is actually asked in practice, but design at least one case per category before skipping any entirely — a category with zero cases is a category with zero test coverage, and rationale/attribution specifically tend to be the ones silently skipped.
Where do the scenarios in each case come from — do we need real client data?
No — scenarios can be entirely synthetic, and should be for anything that will run in CI or a shared test environment. A synthetic decision corpus built around this same five-category task shape can supply ready-made cases if designing every one from scratch isn't practical.
How is this different from just writing unit tests for the agent?
A unit test typically checks one function's output. These cases check the agent's end-to-end memory behavior — did it retrieve the right prior decision and answer correctly from it — which usually spans the retrieval layer and the reasoning layer together, not a single isolated function.
What does a low coverage index actually tell us?
Which specific question shape your team hasn't designed test cases for yet — not how good the agent is. A low index with strong per-case pass rates just means the test suite itself is thin in some categories, which is exactly what this worksheet is meant to surface before it becomes a production surprise.