Rule-Grounding Eval Pack (TY2026)
When we ran four frontier models against 75 current-year planning questions each, 29% of figure-bearing answers quoted a stale number — last year's IRA limit, a superseded Medicare premium, an estate exclusion from before the law changed. Not vague answers: confidently wrong ones, off by exactly one legislative cycle. The Rule-Grounding Eval Pack is the instrument that measures this failure mode. 156 tasks over tax-year-2026 figures, each with the correct value cited to its source document and — this is the part nobody else ships — the enumerated wrong answers a model is likely to give, each tagged with why it's wrong and which year it leaked from.
Preview a task
A real task from the free 50-task open split — the commercial tasks in this pack follow the same contract. The answer key is the product: the correct figure cited to the document that establishes it, and every wrong-but-plausible stale value enumerated with its reason code, so scoring is mechanical.
"user": "As of 2026-09-25. Jurisdiction: US. The client, age 47, participates in her employer's 401(k). What is the 2026 elective deferral limit that applies to her (no catch-up),…" "answer_key": { "expected": 24500, "required": [{ 24,500, "Notice 2025-67" }], "forbidden": [23,500 · prior_year_value (2025), 23,000 · prior_year_value (2024)] }
What ships in the ZIP
Plus the AI Eval Sets methodology PDF, your license certificate, and receipt as separate account downloads. Data ZIP downloads are limited to 3 per order.
Why these tasks discriminate
On this pack's task families, bare frontier systems scored Stale Figure Rates of 0.07–0.63. The same systems with a live figures feed: 0.00 — and the failures that remained were reasoning errors, which is exactly what these tasks catch.
Pre-registered protocol, k=3, published substitution log — see the scoreboard.
Run your evaluation
- 1Present each blinded task to your system, three attempts each.
- 2Collect answers as JSON —
{task_id, attempt, text}. - 3Run
npx tsx score.ts tasks.jsonl answers.json --k 3. - 4Report pass@1, pass^3, and SFR — with the denominator and vintage.
This pack measures the stale-figure gap. The live feed is what closed it in our measurement. Buy the ruler; subscribe to the fix.
Why this Data Set exists
Every model shipping today was trained before most 2026 planning figures were published. The IRS announces retirement limits in November (Notice 2025-67), inflation adjustments in October (Rev. Proc. 2025-32), and CMS sets Medicare premiums in the fall. A model whose training data ends mid-year doesn't know these numbers — but it does know last year's, and it will state them fluently, with full confidence, to a user who has no way to tell the difference.
If you're deploying AI anywhere near financial guidance — a planning copilot, an advisor assistant, a customer-facing chatbot — “how often does it quote last year's limits?” is a question your compliance function will eventually ask, and “we spot-checked a few prompts” is not an answer that survives review. You need a number: a measured rate, on a documented task set, with grading a reviewer can re-run.
That's what this pack produces. Each task asks for a specific 2026 figure a real client conversation would need. The answer key carries the correct value with its primary-source citation, plus a forbidden-figure table: the 2025 value, the 2024 value, superseded and derived-but-never-published variants — each with a reason code. The scorer checks a model's answer against both lists mechanically. The output is a Stale Figure Rate: stale-hit attempts divided by figure-bearing attempts. One number, defensible, reproducible.
Use Cases
Who uses this Data Set
Runs the pack as a pre-deployment gate: no model or prompt change ships to the planning copilot until its Stale Figure Rate on the current vintage is at or below the team's threshold. The scored run goes in the release record.
Cites a scored, documented evaluation — task set, answer keys, grading method, date — in the AI-tool review file instead of a vendor's accuracy claim. Re-runs the same tasks after every vendor model upgrade.
Wires the JSONL tasks into an existing eval harness in an afternoon; the schema is explicit (prompt, answer key, grading method, k-runs) and scoring is mechanical, so results are stable across CI runs with no judge-model variance.
Puts two candidate AI-advice vendors on identical tasks with identical keys and compares Stale Figure Rates head-to-head — the comparison the vendors' own demo environments are designed to prevent.
Measures the before/after of grounding: bare-model SFR versus SFR with the reference feed in context. In our measured contrast run, that delta was 29% stale to 0.00 across every model tested.
What's inside
156 rule-grounding tasks in JSONL: 121 from the commercial split plus the 35-task public rule-grounding split with its keys, so your harness scores the open tasks with the same machinery. Every task carries the question a client conversation would raise, the expected numeric answer, the required figures cited to their source documents (Notice 2025-67, Rev. Proc. 2025-32, the CMS and SSA 2026 announcements), and the forbidden-figure table — every stale or fabricated value we enumerate, each with a reason code (prior_year_value, superseded, wrong_base, fabricated) and the vintage it leaked from.
The pack includes the primary-source-verified figure tables for 2026 and the prior years the forbidden entries reference, the mechanical scorer (a single TypeScript file, no dependencies beyond tsx), and the Methodology PDF documenting task construction, split design, and the pilot of record. Your license certificate and receipt are attached to the delivery email and available in your account portal.
Everything is generated deterministically: same seed and version produce a byte-identical corpus, and every task carries its evals_version and seed in provenance. The 70 held-out tasks — including all 40 tasks used in the published pilot — are never sold in this or any pack, so a future scored comparison against your deployed system stays meaningful.
Technical Highlights
Sample Schema Fields
{
"task_id": <value>,
"family": <value>,
"difficulty": <value>,
"prompt.user": <value>,
"answer_key.expected_numeric": <value>,
"answer_key.required_figures[].source_doc": <value>,
"answer_key.forbidden_figures[].reason": <value>,
"provenance.fact_keys[]": <value>
}Sample queries
The included scorer grades an answers file against the keys mechanically and reports exact-match accuracy plus SFR with the stale-hit count.
npx tsx score.ts tasks.jsonl answers.json --k 3 # → accuracy 0.84 · SFR 0.07 (5/75 figure-bearing attempts stale)
Tasks carry provenance.fact_keys, so you can isolate the figures your product actually surfaces — say, IRA limits — and gate on those alone.
tasks.filter(t =>
t.provenance.fact_keys.includes("retirement.ira.limit.under_50")
)The forbidden-figure tables are data, not prose. Group them by reason code to see exactly which failure class your model hits most.
tasks.flatMap(t => t.answer_key.forbidden_figures)
.reduce((acc, f) => {
acc[f.reason] = (acc[f.reason] ?? 0) + 1;
return acc;
}, {})
// { prior_year_value: ..., superseded: ..., wrong_base: ..., fabricated: ... }Because grading is mechanical and keys are fixed, two runs differ only in the model under test — diff the per-task results directly.
taskIds.map(id => ({
id,
a: runA[id].stale_hit,
b: runB[id].stale_hit
})).filter(r => r.a !== r.b)Methodology
Every figure in the answer keys is verified against its primary source — IRS notices and revenue procedures, SSA and CMS annual announcements — and carries the source document in the key itself. Tasks are generated deterministically from templates over 36 verified fact keys; the generator emits the corpus, the splits, and the benchmark file from one run, so the storefront, the scoreboard, and the shipped pack can never disagree. Grading is mechanical throughout: numeric extraction and matching against required and forbidden tables, k=3 runs per task, no LLM judge anywhere in the path. The pilot of record (four models, 300 scored attempts, published on the scoreboard) was run on 40 tasks that are permanently held out from every product. If a figure in the purchased vintage is corrected within 180 days — an agency revision, a legislative change, an error on our side — we send the corrected pack free.
Frequently asked questions
What exactly is the Stale Figure Rate?+
Stale-hit attempts divided by figure-bearing attempts. An attempt is a stale hit when the model's answer matches an entry in the task's forbidden-figure table — for example, quoting the 2025 IRA limit when the task asks about 2026. It isolates the one-year-behind failure mode from ordinary wrong answers, which is the failure mode that matters for deployed advice systems: a stale figure is fluent, plausible, and wrong in a way users can't detect.
Why mechanical scoring instead of an LLM judge?+
Because the ground truth is numeric and enumerable, a judge model would only add variance and a dependency on someone else's model behavior. Mechanical grading means two people running the same answers file get the same score, which is what makes the result citable in a compliance file. There is no LLM anywhere in the grading path.
What happens when the figures roll in January?+
Each tax-year vintage is a separate SKU — this pack is TY2026 and says so in the name. When the IRS and CMS publish the 2027 figures, we generate and verify the TY2027 vintage as a new pack. Your TY2026 pack remains valid for scoring against 2026 — which stays relevant as long as systems answer 2026-tax-year questions — and corrections to your purchased vintage within 180 days ship free.
How do I know your answer keys are right?+
Every required figure carries its source document in the key (e.g. Notice 2025-67 for retirement limits, Rev. Proc. 2025-32 for inflation adjustments), and the pack includes the verified figure tables so you can check any value against the primary source yourself. The methodology PDF documents the verification pass. If we're wrong, the 180-day correction policy applies and we'd rather hear about it than not.
Are the 35 public tasks the same ones on the free benchmark?+
Yes — the open rule-grounding split is published, keys included, so anyone can validate the task style and grading before buying. The 121 commercial tasks follow the same schema and difficulty mix but are never published. The 70 held-out tasks (including all 40 pilot tasks) are in neither set and are never sold to anyone.
Can we train or fine-tune on these tasks?+
No. The license permits evaluation use only — training use would destroy the pack's value as a measurement instrument, for you and for every other buyer. Every task carries training_use_permitted: false in its license block, and the restriction is in the license certificate that ships with your order.
What did your own pilot actually measure?+
Four models (Claude Sonnet, Gemini, DeepSeek, Haiku), 75 figure-bearing attempts each, on 40 held-out tasks: a 29% aggregate stale rate bare, ranging from 0.07 for the best model to 0.63 for the weakest. With the Rule Sets reference feed in context, the stale rate went to 0.00 for every model tested. Full per-model tables, prompts, and the pre-registered protocol are on the public scoreboard and methodology page.
How does this differ from the Threshold & Cliff pack (EV02)?+
This pack measures whether a model knows the current figure. EV02 measures whether it behaves correctly at the boundaries — where using last year's phase-out range flips the answer from a partial Roth contribution to zero, or one dollar of income triggers a full Medicare surcharge tier. Different failure mode, different task construction. Teams gating a production advice system typically run both; the Full Corpus (EV00) bundles them with decision-recall and picks up future vintages as they ship.
Related Wealth Data Sets
95 threshold tasks where the discontinuity is the point: IRMAA tiers with the two-year lookback, Roth MAGI phase-outs where using the 2025 range flips the answer from a partial contribution to $0, the unindexed NIIT line, the OBBBA $15M estate exclusion. The 36-task adversarial split places every scenario near a boundary so a stale table changes the categorical answer, not just a number. Keys, boundary tables, and mechanical scorer included.
Every sellable task in the AI Eval Sets line, all vintages to date: 401 tasks across rule-grounding (156), threshold-cliff (95, incl. the 36-task adversarial flip family), and decision-recall (150), with answer keys, forbidden-figure tables, boundary tables, and the mechanical scorer. Held-out-split exclusions are documented in the manifest: the 70 held-out tasks — including all 40 pilot tasks — are never sold, so a future scored comparison against your deployed system stays meaningful.
150 retrieval-mode decision-recall tasks over DecisionSynth advisor-decision episodes: given a household's decision history, what was decided, why, what the override reason was, and how it turned out — with answer keys emitted by the episode generator itself. Tasks ship in both in-context and retrieval delivery modes, so you can score a bare model, a RAG pipeline, or an agent-memory product on the same keys. Disjoint from the free DecisionSynth Bench dev set.
Purchases are for internal use only. Redistribution, resale, posting online, or sharing outside your organization is prohibited under the WealthSchema Data License.
Digital product, delivered electronically. All sales are final — no returns, refunds, or exchanges. Defective or incomplete deliveries are corrected free of charge. Data ZIP downloads are limited to 3 per order; account access, your license certificate, and receipt remain available.
Evaluation product: no training use permitted. Corrections to the purchased tax-year vintage within 180 days are provided free.
View data license →