Playbook: Executing SR 11-7 Model Validation Using Synthetic Data

Independence: the validation team's data isn't the development team's data

SR 11-7 emphasizes effective challenge — the validation team is independent of the development team. A common gap: the validation team uses the same training data the development team used, which means the validation discovers only the bugs the dev team's data didn't surface.

Using a synthetic-data corpus that the validation team controls — independent from the development data — is one effective-challenge pattern. The corpus structurally tests scenarios the development team may not have considered. For wealth-tech, this often catches edge cases like ITIN filers, multi-state residents, and K-1 income recipients whose absence from training data hides model defects.

Pillar 1 — Conceptual soundness

Conceptual soundness asks: is the model designed to do what it claims to do? The validation must include: documentation of the model's theoretical underpinnings, an evaluation of the assumptions, an evaluation of the data inputs, and an assessment of the modeling methodology.

The synthetic corpus contributes here by providing a counterfactual test bed. For each major assumption in the model, construct a synthetic scenario that violates the assumption and observe model behavior. If the model fails gracefully, the assumption is bounded; if the model produces nonsense, the assumption is load-bearing and the documentation must reflect that.

Pillar 2 — Ongoing monitoring

Ongoing monitoring tests whether the model continues to perform as expected post-deployment. The validation must include: defined performance metrics, defined thresholds, monitoring frequency, and escalation procedures.

The synthetic corpus contributes by providing the baseline against which production drift is measured. Periodically (typically quarterly) run the model against the same synthetic corpus and compare results to baseline. Drift in the model's outputs against the constant corpus indicates the model itself has drifted or its dependencies (libraries, infrastructure, data pipelines) have changed.

// Quarterly drift check pseudocode
const baseline = JSON.parse(fs.readFileSync('baseline_outputs.json'));
const current = corpus.map(h => model.score(h));

for (const [householdId, baselineOutput] of Object.entries(baseline)) {
  const currentOutput = current[householdId];
  const drift = Math.abs(currentOutput.score - baselineOutput.score);
  if (drift > drift_threshold) {
    flag('drift', householdId, baselineOutput, currentOutput);
  }
}

// If drift count exceeds population threshold, escalate
if (driftCount / corpus.length > 0.05) {
  escalateToModelRiskCommittee();
}

Pillar 3 — Outcomes analysis

Outcomes analysis evaluates the model's predictions against actual outcomes. The validation must include: backtesting against known outcomes, sensitivity analysis, and benchmarking against alternative models.

For models that lack rich actual-outcome data (a new credit model, a new fraud detection model), the synthetic corpus provides synthetic outcomes against which to backtest. The synthetic outcomes must be plausible and structurally consistent — a corpus that includes a 'fraud' label per household, with the label structurally tied to features that should drive the prediction, allows backtesting before real-world deployment.

Effective challenge: the validation team's adversarial role

SR 11-7's effective-challenge principle says the validation team should attempt to find the model's failure modes, not merely confirm what the development team already showed. Using synthetic data, this becomes tractable.

The pattern: the validation team identifies hypotheses about the model's failure modes and constructs synthetic scenarios to test them. Examples for wealth-tech: 'the model fails on households with K-1 income because the training data underrepresented K-1s,' 'the model fails on multi-state residents because the training data was single-state,' 'the model produces inconsistent recommendations for matched pairs differing only in protected class.'

For each hypothesis, construct synthetic households that exercise it and observe model behavior. Document hypotheses confirmed and refuted. The validation report includes both — confirmed failures (with required remediation) and refuted hypotheses (which strengthen the validation conclusion).

Documentation: the validation report structure

The validation report has a standard structure that examiners and model risk committees expect. Following this structure shortens review time.

Validation refresh cadence

Annual revalidation is the standard for production models. Trigger events that force earlier revalidation: material model change, material data input change, performance threshold breach in monitoring, regulatory guidance change, business-context change (new product line, new customer segment).

The revalidation can be lighter than the initial validation if no triggers fired — a focused review of monitoring results, a re-run of the synthetic corpus, and a refresh of the documentation. If triggers fired, the revalidation depth must match the change material.

Key takeaways

• Validation team independence is operationalized through validation-team-controlled synthetic corpora that the development team didn't see during model development.
• Each SR 11-7 pillar — conceptual soundness, ongoing monitoring, outcomes analysis — has a synthetic-data testing pattern that produces evidence the model risk committee and examiners consume.
• Effective challenge is the underrated pillar. Construct hypotheses about model failure modes and test each with targeted synthetic scenarios. Document confirmed and refuted hypotheses both.
• Quarterly drift testing against a constant synthetic baseline is the cheapest signal for whether the model or its dependencies have drifted post-deployment.

FAQ