Buy vs. Build: Synthetic Decision Episodes vs. Mining Your Own CRM
Every team building or evaluating an advisor-facing AI agent eventually asks the same question: where does the training or evaluation data come from? The obvious answer — mine the firm's own CRM notes, case files, and conversation logs — is also the one that turns out to be a multi-month data-engineering project before it produces a single usable example. The alternative is a purchased corpus of synthetic decision episodes: structured, zero-PII records with a ground-truth answer key emitted at generation time. This comparison is about that specific tradeoff — training or evaluation data for a decision-shaped agent-memory system, not the broader anonymized-vs-synthetic question for general wealth-management test data (covered separately).
The two options
Buy: Synthetic Decision Episodes
Deterministically generated decision records — trigger, options considered, recommendation, override and its reason, outcome, cited rule — with QA answer keys written by the generator itself, purchased as a one-time corpus.
- Zero PII by construction — no real client ever appears in the corpus, so there's no consent posture, data-use agreement, or re-identification review to clear before the data can touch a system still being tested
- Every QA task ships with a ground-truth answer key emitted by the generator at creation time — known-answer by construction, with no separate hand-labeling pass and no risk that 'what should have happened' is itself just someone's best guess after the fact
- Archetype and decision-type coverage is a generation parameter, not a hope — each decision pack derives from the same client archetype set as its parent Wealth Data Set bundle, so the patterns worth testing don't depend on whether they happened to occur often enough in one firm's real book
- One-time cost from $495 per pack to $4,950 for the full corpus, delivered immediately as a self-contained ZIP (episodes, QA tasks, households, data dictionary, license, methodology white paper) — no extraction or labeling engineering to build first
- Advisor-behavior parameters — override rates, reason mixes, outcome splits — are authored defaults chosen for scenario diversity, not calibrated to any real advisor population, because no public benchmark for advisor override behavior exists to calibrate against
- Won't reproduce one firm's specific CRM conventions, product shelf, or house policy language verbatim — it's built around typical financial-advisory decision structure, not any single firm's idiosyncratic book
- The same schema and archetype shape ships release to release — a firm with a genuinely unusual client base may still want some real-data validation layered on top
Choose synthetic decision episodes when the goal is evaluating or seeding an agent-memory system's structural capability — can it retrieve rationale, find precedent, order events, cite the governing rule — without spending engineering time on extraction and de-identification first, and without exposing real client data to a system that's still being tested.
Build: Mining Your Own CRM / Interaction Logs
Extracting decision-shaped training or evaluation data from a firm's own case notes, CRM entries, or conversation logs — real interactions and real outcomes, turned into something structured through a data-engineering project the firm builds itself.
- Reflects the firm's actual client base, product mix, and house-specific decision patterns exactly, rather than an approximation of typical advisory structure
- Captures real advisor behavior — the override rates and reasoning a synthetic corpus explicitly does not claim to calibrate to
- No purchase cost for the raw material — the notes and logs already exist inside the firm's own systems
- Every record is real client NPI by definition, which means legal sign-off, a data-use or consent posture, or a de-identification pass (with its own re-identification risk) has to clear before the data can touch a system under test — including a system whose behavior under test is exactly what's uncertain
- No ground truth exists to extract — a CRM note records what happened, not a verified answer to 'what should a memory system retrieve here,' so a labeling step still has to happen after the fact, with the cost and disagreement risk any hand-labeling process carries
- The decisions most worth testing — overrides, escalations, a specific rule citation — are, by definition, the rare ones in any single firm's real history, so a log corpus large enough to test rare-pattern retrieval well can take a long time to accumulate
- Unstructured by default — a case note rarely records 'override reason' or 'cited rule' as its own field, so the extraction pipeline that turns free text into typed, evaluable structure has to be designed and maintained before the data is usable for the same task types a decision-episode schema already ships with
Choose mining the firm's own CRM or logs when the specific goal is capturing that firm's real advisor behavior and client mix for internal calibration or research — not for structural agent-memory evaluation, where the missing ground truth and the PII-review overhead are costs a synthetic corpus simply doesn't carry.
Decision framework
The right choice follows directly from what question is actually being asked. If the question is about the memory system's design — does its pipeline preserve rationale, precedent, temporal order, and rule citations under whatever compression it applies — that's a question about the system, not about any one firm's real behavior, and synthetic episodes with known answers answer it immediately, without the months of extraction and labeling engineering CRM mining requires before a single usable QA pair exists.
If the question is about this specific firm's real advisor behavior — actual override rates, actual reasoning patterns, the product mix particular to this book — only the firm's own real data can answer that, and no synthetic corpus claims otherwise. DecisionSynth's behavioral parameters are deliberately labeled authored, not calibrated, precisely so this distinction never gets blurred into an implicit claim the corpus doesn't back.
Most teams don't actually have to pick one permanently. The synthetic corpus is the fast, zero-risk path to structural evaluation and cold-start seeding today; mining the firm's own interaction history remains the right, slower-track project for behavioral research once legal and data-governance review can be scheduled properly — the two answer different questions and can run in parallel rather than competing for the same budget line.
Bottom line
For agent-memory structural evaluation and cold-start seeding specifically, buying synthetic decision episodes is close to strictly better: the PII-review overhead disappears, ground truth arrives with the data instead of requiring a separate labeling project, and coverage of rare decision types is a design parameter instead of a multi-year accumulation problem. Mining a firm's own CRM remains the right — and only — tool for a narrower, different question: what does this specific firm's real advisor behavior actually look like, which a synthetic corpus deliberately does not claim to answer.
FAQ
Can synthetic decision episodes replace a firm's own behavioral research entirely?+
No, and the corpus says so explicitly. Advisor override rates and reason mixes are authored defaults chosen for scenario diversity, not calibrated to real advisor behavior — there's no public benchmark for that to calibrate against, so claiming calibration would be false. They're the right tool for structural memory evaluation, not a substitute for behavioral research about a specific firm.
What actually makes mining CRM data slow, beyond the extraction code itself?+
Three sequential dependencies most teams underestimate going in: legal sign-off on using real client data for this purpose at all, a de-identification or consent pathway that holds up under review, and a labeling process to produce ground truth that doesn't exist anywhere in the raw notes. Each is real engineering and review time layered on top of the extraction pipeline, not a side detail.
Is the purchased corpus the same data used in the free benchmark?+
No. The commercial corpus is generated at seeds disjoint from both the public dev set and the private held-out test set, asserted disjoint at build time. What you buy is never given away, and buying it doesn't let a system imitate its way onto the benchmark scoreboard.
What exactly ships in a purchased decision pack?+
Each pack is self-contained: episodes.json (the decision episodes), qa.json (evaluation tasks with answer keys), a households/ directory (the fresh synthetic households the episodes derive from, one file per archetype with its 96-month trajectory), a data dictionary, and the license. Every order also includes the DecisionSynth methodology white paper as a second download.
Does buying decision episodes require a legal or privacy review before use?+
Materially less than working with real client data does. No real individuals are referenced anywhere in the corpus, so GLBA, GDPR, and CCPA data-use obligations that attach to real client records don't apply. The license itself is a one-time purchase, perpetual for internal use — development, testing, evaluation, and demonstration inside the purchasing organization.