The discipline of using synthetic data well is harder than the discipline of generating it. Most teams generate a credible-looking corpus and then stop iterating, treating the corpus as static infrastructure rather than as a living asset that should evolve with the product. The result is a corpus that worked at year one and is increasingly mismatched to production reality at year three.
The eight mistakes below are the patterns we see most often in reviews of fintech synthetic-data programs. Each is named, each ships a specific class of production bug, and each has a known remediation pattern. The list is ordered roughly by how often it shows up in incident postmortems we've audited.
1. Field-level realism without joint-distribution realism
The first mistake is the most common. Each individual field looks plausible — names from a name corpus, ages from an age distribution, incomes from an income distribution — but the joint distribution across fields is wrong. The corpus has 25-year-old chief-financial-officers, software engineers in the hospitality industry, and 65-year-olds with two-year-old dependents.
Bug class: ML models trained on the corpus learn spurious correlations that don't reflect real production. Recommendation engines built against the corpus produce shaped-by-noise advice that fails on real customers.
Remediation: validate the joint distributions explicitly. Plot age × income, occupation × industry, marital-status × dependent-count × age. Compare to BLS / ACS reference distributions. Reject the corpus if joint plausibility checks fail, even when field-level distributions look fine.
2. Modal-only coverage with no edge-case archetypes
The second mistake is shipping a corpus that represents only the modal customer. Every household is W-2 salaried, single-state-resident, married-filing-jointly with two dependents, retired or near-retired, with mainstream account types. The corpus is realistic for the bell curve and absent for the long tail.
Bug class: features pass QA on the corpus and crash on real customers in week one of production. The crashes cluster at the long tail — ITIN filers, multi-state residents, K-1 recipients, ISO exercisers.
Remediation: catalog the edge-case archetypes the product must handle and verify each is present with at least 30 households (statistical-power minimum). Treat the catalog as a versioned artifact that grows with each production incident.
3. Static corpus that doesn't evolve with the product
The third mistake is treating the corpus as installation infrastructure rather than a managed asset. The corpus is loaded in 2024, the product evolves through 2025-2027, and the corpus stays at its 2024 calibration. By 2027, the corpus is increasingly mismatched to what production looks like.
Bug class: tests pass against the static corpus while production exhibits new patterns the corpus doesn't reflect. Test confidence becomes detached from production reliability.
Remediation: schedule annual or semi-annual corpus refresh. Each refresh updates the modal-customer distribution to match current production trends, adds new edge-case archetypes surfaced by the year's incidents, and retires archetypes that are no longer relevant.
4. Determinism failure — same seed produces different output
The fourth mistake is non-deterministic generation. Two runs with the same input produce subtly different outputs because the generation pipeline has unseeded random elements somewhere.
Bug class: regression testing breaks. A test that passes today might fail tomorrow against the same nominal corpus. The team reduces or abandons regression testing because the false-positive rate is too high.
Remediation: enforce deterministic generation as a hard requirement. CI test that generating the corpus twice with the same seed produces bit-identical output. Failure is a blocking bug, not a nice-to-have.
5. Over-fitting to the validation suite
The fifth mistake is allowing the corpus to be tuned against the team's specific validation suite, producing high pass rates that don't generalize. The team fixes generation bugs that the suite caught but doesn't notice the bugs the suite missed.
Bug class: the corpus passes internal validation and fails external scrutiny — buyer technical due diligence, third-party audits, regulator data-quality reviews.
Remediation: maintain a held-out validation set that's not used to tune generation. Periodically validate against the held-out set as an external check. Bring in third-party data-quality reviews on a periodic basis.
6. Missing privacy validation
The sixth mistake is generating data that's structurally synthetic but inadequately tested for privacy. The synthetic records may inadvertently reproduce features of specific real records (because the generator drew from a real-data prior too closely), or be uniquely identifiable under presumed-known-fields attacks.
Bug class: a synthetic record can be reverse-engineered to a real customer. The legal and reputational exposure is severe even if the underlying data is in some abstract sense "synthetic."
Remediation: run Sweeney-style re-identification tests on the corpus — for each presumed-known-fields combination (ZIP + age + income, or LinkedIn-equivalent profile), verify no synthetic record is uniquely identifiable. The validation should be automated and gating.
7. Treating synthetic data as production-quality automatically
The seventh mistake is the inverse of #6 — treating synthetic data as flawless because "it's not real customer data, so what could go wrong?" Synthetic-data bugs ship in production code that consumed the synthetic data: wrong field mappings, wrong type conversions, wrong sign conventions on cash flow.
Bug class: production-code bugs that originated in test-data assumptions ship to customers. The bugs are particularly hard to debug because the team trusted the synthetic data more than they would have trusted manually-curated test data.
Remediation: subject synthetic-data integration to the same code-review and validation rigor as other production-data integration. The fact that the data is synthetic doesn't change the requirement that it be correctly handled.
8. Documentation drift between corpus and production schema
The eighth mistake is allowing documentation to drift between the corpus's schema and the production schema. New fields ship in production, the corpus doesn't update, and the consumers of the corpus encounter "the field doesn't exist in the corpus" issues.
Bug class: tests written against the corpus reference fields that don't exist in the corpus, fail silently, or use placeholder defaults that don't match production behavior.
Remediation: treat the corpus schema as a versioned API. Production schema changes that affect the corpus must update the corpus's schema simultaneously. CI checks that corpus schema is consistent with production schema as of each release.
The compound-value pattern
The teams that get the most value from synthetic data treat it as a continuously-improving asset rather than a one-time procurement. Each production incident that traces to a corpus gap adds to the corpus. Each new product feature drives an update to the corpus's coverage matrix. Each annual refresh recalibrates the modal-customer distribution and brings in the year's accumulated improvements.
This compound-value pattern requires organizational discipline that most fintechs don't initially have. The first year of synthetic-data adoption tends to focus on getting the corpus installed and accepted; the compounding value comes in years 2-5 if the team sustains the discipline.
Teams that ship the eight mistakes above don't compound the value. The corpus stays at year-one quality and increasingly drifts from production reality. Within a few years, the team is back to ad-hoc test data with the synthetic corpus a partially-used artifact in the corner.
Key takeaways
- Joint-distribution realism (field-by-field combinations) matters more than field-level realism. Field-level checks pass; joint plausibility produces the spurious correlations that destroy ML model utility.
- Modal-only coverage produces features that pass QA and crash in week one of production. Edge-case archetype coverage is the cheapest engineering investment for production reliability.
- Determinism is a hard requirement, not nice-to-have. Non-deterministic generation breaks regression testing and the team's confidence in tests detached from production reality follows.
- Privacy validation (Sweeney-style re-identification testing) is a gating check, not optional. Synthetic data that's reversible to real customers carries severe legal and reputational risk.
- Treat the corpus as a continuously-improving asset, not a one-time procurement. Year-2-through-5 compound value depends on annual refresh, incident-driven additions, and schema-version discipline.
Most synthetic-data programs underperform their potential because the team treats the corpus as static. The eight mistakes above are the predictable failure modes. Calibrating against the list — and building the organizational disciplines that prevent each mistake — is what separates synthetic-data programs that compound value from those that decay.