AI Eval Sets Full Corpus
An AI advice system can fail three different ways: it can quote last year's figure, it can put a client on the wrong side of this year's threshold, and it can misremember what was decided for the client last quarter. Each failure mode needs its own instrument, and a team standing up serious pre-deployment evaluation ends up needing all three. The Full Corpus is that suite in one purchase: every sellable task in the AI Eval Sets line — 401 tasks across rule-grounding, threshold-cliff, and decision-recall — with every answer key, forbidden-figure table, boundary table, and the scorer — and as future tax-year vintages ship, they join this SKU, so the corpus a buyer receives only grows.
Preview a task
A real task from the free 50-task open split — the commercial tasks in this pack follow the same contract. The answer key is the product: the correct figure cited to the document that establishes it, and every wrong-but-plausible stale value enumerated with its reason code, so scoring is mechanical.
"user": "As of 2026-03-04. Jurisdiction: US, filing status mfj. Client, age 35, married filing jointly, no IRA contributions yet this year. Expected 2026 MAGI is exactly $220,000.…" "answer_key": { "expected": { "full_contribution_allowed": true, "allowed": 7500 }, "required": [{ 242,000, "Notice 2025-67" }, { 252,000, "Notice 2025-67" }], "forbidden": [236,000 · prior_year_value (2025), 246,000 · prior_year_value (2025), 230,000 · prior_year_value (2024)] }
What ships in the ZIP
Plus the AI Eval Sets methodology PDF, your license certificate, and receipt as separate account downloads. Data ZIP downloads are limited to 3 per order.
Why these tasks discriminate
On this pack's task families, bare frontier systems scored Stale Figure Rates of 0.07–0.63. The same systems with a live figures feed: 0.00 — and the failures that remained were reasoning errors, which is exactly what these tasks catch.
Pre-registered protocol, k=3, published substitution log — see the scoreboard.
Run your evaluation
- 1Present each blinded task to your system, three attempts each.
- 2Collect answers as JSON —
{task_id, attempt, text}. - 3Run
npx tsx score.ts tasks.jsonl answers.json --k 3. - 4Report pass@1, pass^3, and SFR — with the denominator and vintage.
This pack measures the stale-figure gap. The live feed is what closed it in our measurement. Buy the ruler; subscribe to the fix.
Why this Data Set exists
Point evaluations produce point conclusions. A system that aces figure recall can still flip answers at IRMAA boundaries; one that handles cliffs can still fabricate a client decision that never happened. If you're gating a production advice system — or comparing vendors who each cherry-pick the benchmark they look best on — you need one task universe, one grading method, and one report that covers the failure modes that actually reach clients.
The deeper problem is auditability over time. An evaluation you run once is a snapshot; the value compounds when you re-run the same instrument after every model upgrade, prompt change, and retrieval change, and can show a reviewer the trend. That only works if the instrument is stable (deterministic tasks, mechanical grading), if its provenance is documented (versions, seeds, splits), and if a genuinely unseen pool still exists for the day someone claims your system was tuned to the test. This corpus is built around those three properties.
Everything sellable is here: the TY2026 rule-grounding and threshold-cliff vintages and the decision-recall pack. Just as important is what is documented as not here: the 70 held-out tasks — including all 40 tasks behind the published pilot — are excluded by construction, listed in the manifest, and never sold to anyone. That exclusion is what keeps a future scored comparison against your deployed system meaningful, even after your team has worked with every task you own.
Use Cases
Who uses this Data Set
Stands up the firm's standard pre-deployment gate in one procurement: every model, prompt, and vendor change scores against the same 401 tasks, and the per-family lines (figures, cliffs, recall) go in the release record.
Keeps one documented evaluation instrument on file for AI-tool review — task provenance, grading method, held-out exclusions — instead of three vendor claims and a spreadsheet of spot checks.
Runs every candidate — bare models, RAG stacks, memory products — on identical tasks and keys across all three families, so no vendor gets to pick the benchmark they look best on.
Wires one JSONL schema into CI once and gets three regression suites; family and difficulty fields make per-line gating (e.g. adversarial cliffs must be perfect) a filter, not a new harness.
Scores their own system before prospects do, on the same corpus those prospects can buy — and cites the per-family results against the public scoreboard's bare-model baselines.
What's inside
All 401 sellable tasks in one JSONL corpus: 156 rule-grounding tasks over IRS/SSA/CMS tax-year-2026 figures with primary-source citations and forbidden-figure tables; 95 threshold-cliff tasks including the 36-task adversarial split where a stale table flips the categorical answer; and 150 decision-recall tasks over DecisionSynth advisor-decision episodes, which ship in the pack for in-context or retrieval delivery. Every task carries family, difficulty, split, vintage, and full provenance (evals_version, seed, template, fact keys), so the corpus slices cleanly into whatever evaluation lines your gate needs.
Supporting assets: the verified 2026 figure and boundary tables plus the prior-year tables the forbidden entries reference, the mechanical scorer, the Methodology PDF, and the manifest documenting the 70 held-out exclusions — including all 40 pilot tasks — that are never sold in any product. Your license certificate and receipt are attached to the delivery email and available in your account portal.
The rule-grounding and threshold-cliff families are the TY2026 vintage; when the figures roll each January, new vintages ship as new SKUs, and this corpus SKU always contains every vintage to date at purchase time. The decision-recall family isn't vintaged — recorded decisions don't expire. Corrections to your purchased cut within 180 days ship free.
Technical Highlights
Sample Schema Fields
{
"task_id": <value>,
"family": <value>,
"difficulty": <value>,
"split": <value>,
"vintage": <value>,
"answer_key.expected_numeric": <value>,
"answer_key.forbidden_figures[].reason": <value>,
"provenance.evals_version": <value>
}Sample queries
One scorer pass over all 401 tasks; the family field turns the output into the three-line report a deployment gate actually wants.
npx tsx score.ts tasks.jsonl answers.json --k 3 # then aggregate: ["rule_grounding", "threshold_cliff", "decision_recall"] .map(f => summarize(results.filter(r => r.family === f)))
A sane production gate isn't uniform: adversarial cliffs and stale-figure hits are the clauses with dollar costs attached.
const gate = sfr(results) === 0 && pass(results.filter(r => r.difficulty === "adversarial")) && accuracy(results) >= 0.95;
Every task carries version, seed, and template — the audit trail that lets a reviewer reproduce exactly what was tested, months later.
results.map(r => ({
id: r.task_id,
pass: r.pass,
version: tasksById[r.task_id].provenance.evals_version,
seed: tasksById[r.task_id].provenance.seed
}))The manifest enumerates what you did not receive. Confirm your corpus contains no held-out task — the property that keeps future scored comparisons fair.
const heldOut = new Set(manifest.held_out_task_ids); tasks.every(t => !heldOut.has(t.task_id)) // → true
Methodology
The three families share one generation and grading doctrine. Figures are verified against primary sources and cited in the keys; boundary placements are derived from the verified year-over-year tables; decision-recall keys are emitted by the episode generator itself, so ground truth is known by construction rather than labeled. Generation is deterministic — same seed and version, byte-identical corpus, splits included — and every task is version-stamped in provenance. Grading is mechanical throughout (numeric and structured matching, text assertions for recall, k=3 runs), with no LLM judge anywhere in the path. The pilot of record and the feed-in-context contrast run are published on the public scoreboard with their pre-registered protocol; the 40 pilot tasks sit inside the 70-task held-out pool this corpus documents and excludes. Corrections to the purchased cut within 180 days ship free.
Frequently asked questions
What does the Full Corpus cost versus buying the packs separately?+
The three packs total $3,895 separately ($1,450 + $1,450 + $995); the Full Corpus is $3,950 — effectively the three packs plus the all-vintages guarantee: as new tax-year vintages ship, the corpus SKU always contains every vintage to date at purchase time, so a later buyer gets more tasks at the same price. If you only need one failure mode measured, buy that pack; if you're standing up a real gate, you'll want all three families anyway.
What exactly is excluded, and how do I verify it?+
Seventy tasks are held out: 40 rule-grounding and threshold-cliff tasks reserved from the pilot, and 30 decision-recall tasks drawn from a disjoint episode set. They are never published, never sold, and never included in any pack. The manifest in your download enumerates the exclusions so you can verify your corpus against it — and so a reviewer can confirm that any future scored comparison we publish against your system used tasks nobody could have bought.
Why should we trust the scores mean anything after our team has seen every task?+
Working with purchased tasks is expected — that's what a regression suite is. Two properties keep the measurement honest anyway: training on the tasks is prohibited by license (training_use_permitted: false on every task), and the held-out pool exists precisely for the arms-length case. Your internal trend line runs on the corpus you own; a claim that needs to survive outside scrutiny can be checked against tasks that were never for sale.
How do vintages work across the families?+
Rule-grounding and threshold-cliff are vintaged to the tax year, because their ground truth rolls every January — this cut is TY2026, verified against the 2026 source documents. Decision-recall is not vintaged; it tests memory of recorded decisions, which don't expire. When TY2027 figures publish, the new vintage ships as new SKUs and joins this corpus for subsequent buyers; your purchased cut stays valid for 2026-tax-year scoring and receives free corrections for 180 days.
Can one harness really run all three families?+
Yes — the schema is uniform: every task has a prompt, an answer key, a grading block, and provenance. The differences are in the key (numeric figures with forbidden tables, structured categorical results, or text assertions over episodes) and the scorer handles all three. The one integration choice is decision-recall delivery: embed the shipped episodes in-context, or serve them through the memory or retrieval product under test.
How does this relate to the free public benchmark?+
The 50-task open split (35 rule-grounding, 15 threshold-cliff, keys included) is published so anyone can validate task style and grading before spending anything, and the public scoreboard shows measured bare-model results with a pre-registered protocol. This corpus includes those 50 open tasks plus the 351 commercial tasks that never appear anywhere public. Decision-recall has no open split at all.
Can we train or fine-tune on any of this?+
No — the entire line is evaluation-only, and every task carries the restriction in its license block. If you need training or seed data, that's what the Wealth Data Sets and DecisionSynth lines are for; the strict separation between what you train on and what you're graded on is the point of the product.
Related Wealth Data Sets
156 rule-grounding tasks over IRS/SSA/CMS 2026 figures — each with the correct value cited to its source document (Notice 2025-67, Rev. Proc. 2025-32, CMS 2026) and the enumerated wrong-but-plausible answers: the 2025 value, the 2024 value, superseded and derived-not-published variants, each with a reason code. Ships with answer keys, the primary-source-verified prior-year figure tables, and a mechanical scorer. Score any model's Stale Figure Rate in an afternoon; re-buy the vintage when the figures roll.
95 threshold tasks where the discontinuity is the point: IRMAA tiers with the two-year lookback, Roth MAGI phase-outs where using the 2025 range flips the answer from a partial contribution to $0, the unindexed NIIT line, the OBBBA $15M estate exclusion. The 36-task adversarial split places every scenario near a boundary so a stale table changes the categorical answer, not just a number. Keys, boundary tables, and mechanical scorer included.
150 retrieval-mode decision-recall tasks over DecisionSynth advisor-decision episodes: given a household's decision history, what was decided, why, what the override reason was, and how it turned out — with answer keys emitted by the episode generator itself. Tasks ship in both in-context and retrieval delivery modes, so you can score a bare model, a RAG pipeline, or an agent-memory product on the same keys. Disjoint from the free DecisionSynth Bench dev set.
Purchases are for internal use only. Redistribution, resale, posting online, or sharing outside your organization is prohibited under the WealthSchema Data License.
Digital product, delivered electronically. All sales are final — no returns, refunds, or exchanges. Defective or incomplete deliveries are corrected free of charge. Data ZIP downloads are limited to 3 per order; account access, your license certificate, and receipt remain available.
Evaluation product: no training use permitted. Corrections to the purchased tax-year vintage within 180 days are provided free.
View data license →