The whole business rests on answer keys being right. Here is how they are made, attacked, and corrected.

1 · The admission rule

A benchmark item ships only if its answer key can be verified without consulting the open web. Three task classes qualify: deduction from a closed corpus of primary sources (the key is a derivation from a cited paragraph); computation (the key is produced by code, deterministically); and constraint violation against enumerated rules (closed-world by construction). Open-web factual knowledge — market shares, vendor capabilities, "industry best practice" — is inadmissible. This narrows what we test to compliant, arithmetically correct, rule-consistent rather than "wise," and we say so openly: the narrower half is the half regulated buyers are required to care about.

2 · The closed corpus (2026.9)

Every source is fetched from its issuing authority, stored raw, extracted to canonical text, and hashed. Items pin the corpus version they were adjudicated against; a source amendment triggers targeted re-adjudication. The corpus is reviewed quarterly and on every known amendment.

3 · The production protocol

AI drafts everything; the pipeline decides what counts.

1. A generator model drafts scenario, key, derivation, and rubric against a written task-family spec.
2. A mechanical gate checks schema, unique ids, corpus pins, that every cited quote exact-matches the corpus text, and that every computed value equals its own compute function.
3. Adversarial verifiers — two models from vendors other than the generator's — attack each key, blinded among planted controls. A verdict is admissible only if it cites a passage that survives the quote gate. Agreement counts for nothing; disagreement routes the item to a human.
4. Planted controls: each batch is seeded with deliberately wrong keys. If any control goes uncaught, the batch fails and no item ships from it.
5. Human adjudication resolves flagged items by reading the cited paragraphs — never the open web — and signs the resolution by name.

The first production batch (2026-09-01) is a working example, defects included: of 41 blinded units, all 5 planted controls were caught, 28 items verified cleanly, and 8 were flagged — 5 for incomplete table citations and 3 where a verifier read the quoted rule differently. The flagged items were repaired or routed to adjudication; the repairs are visible in each item's ledger. A pipeline that never flags anything is not a pipeline; it is a rubber stamp.

4 · Versioning, errata, and the standing challenge

Answer keys are versioned; corrections are conspicuous. Anyone who demonstrates a key error gets a public, credited erratum on the item's provenance page — the ledger keeps the wrong version visible, what changed, and who found it. Write to benchmark@wealthschema.com with the item id and your derivation from the cited sources. Fast, public correction is the credibility strategy; a survived challenge is an asset, not an incident.

5 · Who pays for what — the full list

A small firm rating a market it also sells services into must over-disclose. The complete set of ways money reaches WealthSchema, and what each buys:

6 · Splits, gaming, and rotation

Public and held-out suites are different datasets with different code paths — nothing filters one out of the other, so a held-out item cannot leak by a forgotten filter. Held-out suites back private runs and are rotated; item sets and the corpus are versioned so every published number names exactly what it was measured against. Training on the public split is detectable by the gap it opens against held-out results, and that gap is itself reported when we see it.