Threshold & Cliff Eval Pack (TY2026)
A model that quotes a stale figure is wrong by a number. A model that applies a stale threshold is wrong by a decision. Tell a client near a Medicare income tier that they're under the line when this year's table says they're over it, and the cost isn't a rounding error — it's a full year of IRMAA surcharge, triggered by advice. The Threshold & Cliff Eval Pack tests exactly this: 95 tasks where the discontinuity is the point, including a 36-task adversarial split where every scenario is placed close enough to a boundary that using last year's table changes the categorical answer — eligible becomes ineligible, partial becomes zero, under becomes over.
Preview a task
A real task from the free 50-task open split — the commercial tasks in this pack follow the same contract. The answer key is the product: the correct figure cited to the document that establishes it, and every wrong-but-plausible stale value enumerated with its reason code, so scoring is mechanical.
"user": "As of 2026-03-04. Jurisdiction: US, filing status mfj. Client, age 35, married filing jointly, no IRA contributions yet this year. Expected 2026 MAGI is exactly $220,000.…" "answer_key": { "expected": { "full_contribution_allowed": true, "allowed": 7500 }, "required": [{ 242,000, "Notice 2025-67" }, { 252,000, "Notice 2025-67" }], "forbidden": [236,000 · prior_year_value (2025), 246,000 · prior_year_value (2025), 230,000 · prior_year_value (2024)] }
What ships in the ZIP
Plus the AI Eval Sets methodology PDF, your license certificate, and receipt as separate account downloads. Data ZIP downloads are limited to 3 per order.
Why these tasks discriminate
On this pack's task families, bare frontier systems scored Stale Figure Rates of 0.07–0.63. The same systems with a live figures feed: 0.00 — and the failures that remained were reasoning errors, which is exactly what these tasks catch.
Pre-registered protocol, k=3, published substitution log — see the scoreboard.
Run your evaluation
- 1Present each blinded task to your system, three attempts each.
- 2Collect answers as JSON —
{task_id, attempt, text}. - 3Run
npx tsx score.ts tasks.jsonl answers.json --k 3. - 4Report pass@1, pass^3, and SFR — with the denominator and vintage.
This pack measures the stale-figure gap. The live feed is what closed it in our measurement. Buy the ruler; subscribe to the fix.
Why this Data Set exists
U.S. planning rules are full of discontinuities that punish being one dollar wrong. IRMAA surcharges apply in full the moment MAGI crosses a tier — and the relevant income is from two years ago, a lookback that models routinely drop. Roth IRA eligibility phases out over a MAGI range that moves every year; run the 2025 range against a 2026 client and a permitted partial contribution becomes an excess-contribution problem. The NIIT thresholds have never been indexed — a model that “helpfully” inflation-adjusts them invents a threshold that doesn't exist. And the estate exclusion changed by statute, not indexing, when OBBBA set it to $15M — a figure no pre-2025 training corpus contains.
Generic accuracy benchmarks miss all of this, because away from the boundary a stale table still produces the right categorical answer most of the time. A model can score well on “what is the Roth phase-out range?” and still fail every client who actually sits inside it. The only way to measure boundary behavior is to construct scenarios at the boundary — which requires knowing, precisely and with sources, where this year's boundary is and where last year's was.
This pack is those scenarios. Each task poses a concrete situation — an income, a filing status, an age — near a 2026 boundary, and asks the question a client would ask. The answer key holds the correct categorical or numeric result, the required 2026 figures with citations, and the forbidden figures: the prior-year boundary values that produce the wrong answer, tagged with reason and vintage. The adversarial split is the discriminating instrument — in our pilot it was the task family where model quality separated most sharply.
Use Cases
Who uses this Data Set
Gates releases on the adversarial split specifically: a system can pass broad figure-recall and still flip answers at boundaries, and the boundary cases are the ones where wrong advice has a dollar cost attached.
Documents that the firm's AI tooling was tested on cliff-sensitive topics — Medicare tiers, Roth eligibility, the estate exclusion — with a scored, re-runnable evaluation rather than vendor assurances.
Adds the pack as a regression suite: threshold behavior is where prompt and model upgrades regress silently, because the top-line accuracy number barely moves when a handful of boundary cases flip.
Validates that an IRMAA or Roth-eligibility feature answers correctly on both sides of every 2026 tier before shipping, using boundary tables verified against the CMS and IRS source documents.
Runs candidate vendors on the 36 adversarial flips head-to-head. Categorical answers make the comparison stark: each vendor either places the client on the correct side of the line or doesn't.
What's inside
95 threshold-cliff tasks in JSONL: 80 commercial plus the 15-task public threshold split with keys. The topic mix follows where cliff exposure actually concentrates: Roth MAGI phase-outs for both filing statuses, IRMAA tiers with the two-year lookback baked into the scenarios, the unindexed NIIT thresholds, the Social Security earnings test for workers under full retirement age, the saver's credit AGI limits, the OBBBA estate exclusion and annual gift exclusion, and IRA and HSA contribution-limit boundaries.
Thirty-six tasks form the adversarial split: scenarios placed near a 2026 boundary such that substituting the 2025 table flips the categorical answer, not just the number. These are tagged difficulty: adversarial so you can score them as a separate line — which we recommend, because it's the line that moves when a system regresses.
The pack ships with the verified 2026 boundary tables and the prior-year tables the forbidden entries reference, the mechanical scorer, and the Methodology PDF. Answer keys use expected_structured for categorical results (eligible / phase-out / ineligible, tier placements) so grading stays mechanical even when the answer isn't a single number. Generation is deterministic and version-stamped; the 70 held-out tasks, including all pilot tasks, are never sold.
Technical Highlights
Sample Schema Fields
{
"task_id": <value>,
"family": <value>,
"difficulty": <value>,
"prompt.user": <value>,
"answer_key.expected_structured": <value>,
"answer_key.required_figures[].key": <value>,
"answer_key.forbidden_figures[].vintage": <value>,
"provenance.template_id": <value>
}Sample queries
The adversarial split is the discriminating instrument — score it as its own line so boundary regressions can't hide inside the aggregate.
npx tsx score.ts tasks.jsonl answers.json --k 3 # then filter per-task results: results.filter(r => tasksById[r.task_id].difficulty === "adversarial")
Isolate a single boundary — say, IRMAA — to test a feature that only touches Medicare tiers.
tasks.filter(t =>
t.provenance.fact_keys.includes("medicare.irmaa")
)Each adversarial task's forbidden figures document which prior-year boundary produces the wrong side of the line, with vintage.
tasks
.filter(t => t.difficulty === "adversarial")
.map(t => ({
id: t.task_id,
correct: t.answer_key.expected_structured,
trap: t.answer_key.forbidden_figures.map(f => `${f.vintage}: ${f.reason}`)
}))IRMAA scenarios state the income year explicitly; a model that applies current-year income to the tier table fails these even with the right 2026 tiers in hand.
tasks.filter(t =>
t.provenance.template_id.startsWith("tc-irmaa")
)Edge cases modeled
The pack's coverage is enumerable, so we enumerate it. Counts are tasks in the sellable TY2026 cut (95 total) whose provenance touches each boundary; a task can touch more than one.
Scenarios inside the 2026 phase-out range for single and MFJ filers where the 2025 range yields a different classification — full contribution instead of partial, or partial instead of zero.
retirement.roth_ira.magi_phaseout.*Medicare surcharge placement using two-years-prior MAGI against the 2026 tier table. Tests both stale tiers and the dropped-lookback error, which produce different wrong answers.
medicare.irmaaThe statutory net-investment-income-tax lines that have never been indexed. The trap runs backwards: models fabricate an inflation-adjusted threshold that has never existed.
tax.niit.magi_threshold.*Benefit-withholding scenarios for workers under full retirement age near the 2026 annual exempt amount, where the stale exempt amount changes the withheld dollars.
social_security.earnings_test.under_fra_annualCredit-rate tier placement near the 2026 AGI breakpoints, where a stale table moves the household between the 50%, 20%, 10%, and 0% tiers.
retirement.savers_credit.agi_limit.mfjEstate scenarios sized between the pre-OBBBA figures and the statutory $15M, where every pre-2025 source — stale figure or sunset-reversion figure — misclassifies the estate.
estate.basic_exclusionContribution scenarios at the 2026 IRA limit and the HSA family and age-55+ catch-up limits, where the stale limit misclassifies a permitted contribution as excess — or waves an excess contribution through.
retirement.ira.limit.under_5036 of the 95 tasks form the adversarial split, where the scenario is placed so a stale table flips the categorical answer. Counts computed from the shipped corpus's provenance fields — the same tasks.jsonl you receive.
Methodology
Boundary placements are derived, not invented: for each cliff we hold the verified 2026 table and the prior-year tables side by side, and the generator places scenario incomes and ages in the interval where the tables disagree — which is precisely where a stale model gives the wrong categorical answer. All figures are verified against primary sources (IRS notices and revenue procedures, the CMS premium announcement, the SSA fact sheet) and cited in the keys. Grading is mechanical: structured-answer matching for categorical results, numeric matching against required and forbidden tables otherwise, k=3 runs per task, no LLM judge. The corpus is generated deterministically with version-stamped provenance, and the pilot of record — including this family's sharp per-model separation — is published on the scoreboard with its pre-registered protocol. Corrections to the purchased vintage within 180 days ship free.
Frequently asked questions
What makes a task “adversarial” here?+
Placement. An adversarial task's scenario sits close enough to a 2026 boundary that substituting the 2025 table flips the categorical answer — a client who is over the IRMAA tier under this year's thresholds but under it with last year's, or inside the Roth phase-out this year but fully eligible on the stale range. Away from boundaries, stale tables usually still give the right answer, which is exactly why non-adversarial benchmarks overestimate deployed systems.
How is a categorical answer graded mechanically?+
The key's expected_structured field holds the exact categorical result — a tier placement, an eligibility status, a phase-out classification plus the resulting dollar amount where applicable. The scorer matches structure and values directly. No judge model interprets the answer; if your harness produces the structured fields, grading is deterministic.
Does the pack really handle the IRMAA two-year lookback?+
Yes — IRMAA scenarios state which year's MAGI is given, and the correct answer requires applying the two-years-prior income to the 2026 tier table. Models fail these two ways: stale tiers, or right tiers applied to the wrong income year. Both are scored wrong, and the forbidden-figure entries distinguish the stale-table case.
Why does an unindexed threshold like NIIT even need testing?+
Because models fail it in the opposite direction. The NIIT thresholds have been fixed by statute since 2013, and a model that has learned “planning figures adjust annually” will sometimes fabricate an inflation-adjusted value. The forbidden tables enumerate these fabricated-forward figures alongside genuinely stale ones — different reason codes, both scored as failures.
Is the estate exclusion really that different in 2026?+
Yes — OBBBA set the basic exclusion to $15M by statute, replacing both the prior-law 2025 figure and the sunset-reversion figure that pre-2025 training data expects. It's the cleanest test in the pack for whether a system's knowledge postdates the legislation, because every pre-OBBBA source gets it wrong in one of two distinct, enumerable ways.
Can we run just this pack, or do we need EV01 too?+
They measure different failure modes: EV01 asks whether the model knows the current figures; this pack asks whether it behaves correctly where the figures bite. A system can pass one and fail the other. If you're gating a production advice system, run both — or take the Full Corpus (EV00), which includes both families plus decision-recall in one purchase.
Can we train or fine-tune on these tasks?+
No — evaluation use only, and every task carries training_use_permitted: false. Boundary tasks are especially cheap to memorize and expensive to re-create honestly, so the restriction is what keeps your future scores meaningful.
Related Wealth Data Sets
156 rule-grounding tasks over IRS/SSA/CMS 2026 figures — each with the correct value cited to its source document (Notice 2025-67, Rev. Proc. 2025-32, CMS 2026) and the enumerated wrong-but-plausible answers: the 2025 value, the 2024 value, superseded and derived-not-published variants, each with a reason code. Ships with answer keys, the primary-source-verified prior-year figure tables, and a mechanical scorer. Score any model's Stale Figure Rate in an afternoon; re-buy the vintage when the figures roll.
Every sellable task in the AI Eval Sets line, all vintages to date: 401 tasks across rule-grounding (156), threshold-cliff (95, incl. the 36-task adversarial flip family), and decision-recall (150), with answer keys, forbidden-figure tables, boundary tables, and the mechanical scorer. Held-out-split exclusions are documented in the manifest: the 70 held-out tasks — including all 40 pilot tasks — are never sold, so a future scored comparison against your deployed system stays meaningful.
150 retrieval-mode decision-recall tasks over DecisionSynth advisor-decision episodes: given a household's decision history, what was decided, why, what the override reason was, and how it turned out — with answer keys emitted by the episode generator itself. Tasks ship in both in-context and retrieval delivery modes, so you can score a bare model, a RAG pipeline, or an agent-memory product on the same keys. Disjoint from the free DecisionSynth Bench dev set.
Purchases are for internal use only. Redistribution, resale, posting online, or sharing outside your organization is prohibited under the WealthSchema Data License.
Digital product, delivered electronically. All sales are final — no returns, refunds, or exchanges. Defective or incomplete deliveries are corrected free of charge. Data ZIP downloads are limited to 3 per order; account access, your license certificate, and receipt remain available.
Evaluation product: no training use permitted. Corrections to the purchased tax-year vintage within 180 days are provided free.
View data license →