Playbook: Quarterly Compliance Dry-Run on a Synthetic Customer Population

Scope: which compliance functions does the dry-run cover

Not every compliance function benefits equally from a synthetic-data dry-run. The functions that benefit most: supervisory review (the engine that flags trades, recommendations, communications for review), exception reporting (the system that identifies anomalies for compliance investigation), and audit-trail generation (the system that produces the artifact regulators consume).

Functions that don't benefit much: anti-money-laundering KYC document review (the documents themselves are the artifact, not the supervisory output), fingerprinting and registration (those happen at hire / onboard, not dynamically). Scope the dry-run to the dynamic-supervisory functions where the engine's behavior on the synthetic population reveals program effectiveness.

Population: what the synthetic corpus must include

The corpus must include households that exercise each supervisory rule. A corpus of 'normal' households doesn't exercise the supervisory engine — it exercises the no-flag-needed path. The dry-run must intentionally seed the corpus with households that should fire each rule.

Execution: running the engine against the population

Load the synthetic population into the supervisory environment (your dev or staging env, with the supervisory engine running its production logic). Generate the activity that triggers supervisory review — synthetic recommendations, synthetic transactions, synthetic communications.

For each household, the supervisory engine should produce a known set of expected outputs: alert / no alert, alert reason, supervisory route, escalation status. The dry-run captures the actual outputs and compares to expected.

// Dry-run execution sketch
const expectedFor = loadExpectedSupervisoryOutputs();
const results = [];

for (const household of synthetic_corpus) {
  const activities = generateActivity(household);
  for (const activity of activities) {
    const supervisoryOutput =
      supervisoryEngine.evaluate(household, activity);
    const expected = expectedFor[household.id][activity.id];
    results.push({
      household_id: household.id,
      activity_id: activity.id,
      expected,
      actual: supervisoryOutput,
      match: deepEqual(expected, supervisoryOutput)
    });
  }
}

const mismatches = results.filter(r => !r.match);
generateDryRunReport(results, mismatches);

Evaluation: classifying the gaps

Mismatches between expected and actual outputs fall into four classes: (1) false negatives — the engine should have flagged but didn't; (2) false positives — the engine flagged when it shouldn't; (3) wrong-route — the engine flagged but routed to the wrong reviewer; (4) missing-rationale — the engine produced an output but without sufficient documentation.

False negatives are the highest-priority gaps — these are the behaviors that would fail an examination. False positives are lower priority but have operational cost (wasted review time). Wrong-route and missing-rationale are documentation-quality issues that affect examination but not customer outcomes directly.

Documentation: the dry-run report

The dry-run report is the artifact for the compliance committee and (under examination) for the regulator. The report structure: executive summary of findings; per-rule breakdown of expected vs. actual; classification of each mismatch (false negative / positive / wrong-route / missing-rationale); remediation plan with target dates; comparison to prior quarter's dry-run trends.

A single quarter is a snapshot; a year of quarterly dry-runs shows program maturation or regression. Regulators view a regression trend more skeptically than a single-quarter finding.

Remediation tracking

Each gap surfaced by the dry-run gets entered in the program's remediation tracker with: gap description, severity classification, owner, target completion date. The tracker rolls forward across quarters — a gap not closed within the quarter rolls to next quarter with explicit reasoning for the deferral.

The tracker becomes its own examination artifact. A program that surfaces gaps and closes them quickly is in better shape than a program that doesn't surface them at all.

Cadence: why quarterly works

Monthly dry-runs are too frequent — the cycle of execution + evaluation + remediation can't reliably complete in a month. Annual dry-runs are too infrequent — gaps fester for too long before discovery.

Quarterly hits the sweet spot: the dry-run takes 1-2 weeks to execute and evaluate, followed by 6-8 weeks of remediation, followed by the next quarter's dry-run validating the remediation. Synthetic corpora can be refreshed quarterly to introduce new test scenarios — the corpus itself is a tool the compliance team can evolve.

Key takeaways

• The cheapest place to find a compliance program gap is your own quarterly dry-run on synthetic data — orders of magnitude cheaper than discovering the gap from a regulator or a customer complaint.
• The corpus must be intentionally seeded with households that exercise each supervisory rule. A 'normal' corpus exercises the no-flag-needed path, not the engine.
• False negatives — the engine missed a flag — are the highest-priority gaps. These are the behaviors that fail examinations and produce customer-harm scenarios.
• Quarterly is the right cadence — frequent enough that gaps don't fester, infrequent enough that the cycle of dry-run, evaluation, and remediation can complete cleanly each round.

FAQ