The Planning Benchmark Pilot of Record: Protocol, Scoring, and Results
In July 2026 we ran a pre-registered evaluation of four AI systems from three labs on 40 routine U.S. financial-planning questions, with no tools and no retrieval, three attempts per system. This page is the complete protocol and the complete result — the decision rule frozen before any model was called, what counted as a pass, what went wrong in the harness and how it was handled, and the numbers with their denominators. Every benchmark figure published anywhere on this site cites this page.
What was measured, on which systems
Four systems, three labs, frozen before execution: claude-sonnet and claude-haiku (Anthropic, run as session aliases — the exact dated model strings were not exposed by the harness, a recorded limitation), gemini-3.1-pro-preview (Google, a preview endpoint that may be updated server-side; risk accepted and recorded), and deepseek-v4-pro (DeepSeek).
All systems ran at provider defaults with no tools and no retrieval, max_tokens 8192, k=3 stateless attempts each. The 40-task grid produced 480 task-attempts, all collected and scored. Tasks spanned four families: rule-grounding (15), threshold/cliff (10), in-context memory fidelity (10), and override justification (5).
- ·Answer-key authority: 2026 figures from the Rule Sets feed (status official_final); wrong-but-plausible prior-year figures from dedicated 2025/2024 tables, every fact-year verified against its primary government document (150/150 verified before execution).
- ·Declared deviation: all 40 tasks were presented in one batched prompt per (system, attempt) rather than per-task calls — identical file for every system, so within-attempt independence is imperfect for all systems equally.
- ·The one system-with-tools contrast run (same model with a live Rule Sets lookup) was explicitly out of scope for the pilot and is deferred to a follow-up.
The definition of a pass (frozen before execution)
An attempt passes a task if and only if: (a) every required figure is stated with the correct value for the task's as-of date — exact for limits and thresholds, per-task tolerance for computed values — with source attribution where the task demands it; (b) no forbidden figure appears as the operative value (a prior-year value correctly labeled as prior-year is not a hit); (c) required assertions are present; and (d) no forbidden assertion is present.
Hedged-but-committed answers pass on the committed value. Dual uncommitted values fail. Mechanical fields (figure values, forbidden hits, fabricated citations) were scored by script; semantic assertion fields second, with a blind second-scorer pass over all failed-or-marginal attempts and every reconciliation logged.
The Stale Figure Rate
The Stale Figure Rate (SFR) is the share of figure-bearing attempts in which the system asserted at least one forbidden figure — a stale, superseded, or fabricated regulatory value — as current. The denominator is attempts on forbidden-defining tasks (all rule-grounding and threshold/cliff tasks: 75 attempts per system).
Reason codes, frozen in the pre-registration: prior_year_value, superseded, wrong_tier_or_status, wrong_base, fabricated, derived_not_published. The last of these — computing a plausible figure from an indexing rule or stale base instead of recalling the published value — was added to the taxonomy when the pilot surfaced it as the signature frontier-model failure mode.
The pre-registered decision rule, and the result against it
The GO/NO-GO rule was frozen before any system call. GO required all three conditions; all three were met.
Headline result, pre-registered sentence with the numbers filled in: asked 40 routine 2026 U.S. planning questions with no tools, four systems from three labs applied a stale or fabricated regulatory figure in 29% of figure-bearing attempts (86 of 300), and only 85% of tasks were answered correctly on all three tries by the best system — the other three managed 33–42%.
- ·Condition 1 — rule-grounding + threshold/cliff pass@1 aggregate ≤ 0.65: measured 0.387. Met.
- ·Condition 2 — SFR ≥ 0.20 on ≥ 2 of 4 systems: claude-haiku 0.63, deepseek-v4-pro 0.28 (gemini-3.1-pro-preview 0.17 just misses). Met.
- ·Condition 3 — ≥ 1 named failure mode in ≥ 3 of 4 systems: two qualified (prior_year_value operative in three systems; derived_not_published in three systems). Met.
Results by system (pass@1, pass^3, Stale Figure Rate)
Per-system results over the full 40-task grid at k=3. pass@1 is the first-attempt pass rate with a 95% confidence interval; pass^3 requires all three attempts to pass; rg+tc pass@1 restricts to the rule-grounding and threshold/cliff families; SFR is stale hits over 75 figure-bearing attempts.
claude-sonnet: pass@1 0.90 [0.83–0.94], pass^3 0.85, rg+tc 0.84, SFR 0.07 (5/75).
claude-haiku: pass@1 0.36 [0.28–0.45], pass^3 0.33, rg+tc 0.11, SFR 0.63 (47/75).
gemini-3.1-pro-preview: pass@1 0.57 [0.48–0.65], pass^3 0.42, rg+tc 0.33, SFR 0.17 (13/75).
deepseek-v4-pro: pass@1 0.48 [0.40–0.57], pass^3 0.35, rg+tc 0.27, SFR 0.28 (21/75).
Floor-task pass^3 — tasks a competent system must never miss, passed on every attempt: claude-sonnet 1.00, gemini-3.1-pro-preview 0.50, deepseek-v4-pro 0.40, claude-haiku 0.30. In-context memory fidelity was saturated at 1.00 for all four systems across 120 attempts with genuine same-type distractors — no memory failure mode was found.
The failure taxonomy (what the wrong answers actually were)
Wrong answers were fluent, cited, and repeated near-identically across attempts — pass^3 approximately equals pass@1 on every system, so the errors are systematic properties of the systems, not sampling noise.
- ·prior_year_value — a superseded published figure asserted as current fact; dominant for claude-haiku (2024 and 2025 values asserted as 2026 fact) and material for deepseek-v4-pro and gemini-3.1-pro-preview.
- ·derived_not_published — correct statute, correct method, wrong figure: computing '150% × $8,000 = $12,000' for a catch-up limit published at $11,250, or extrapolating an earnings-test threshold from the prior year instead of recalling the published value. Observed in claude-sonnet, gemini-3.1-pro-preview, and deepseek-v4-pro.
- ·superseded_law_missed — two systems asserted the repealed TCJA estate-tax sunset (a roughly $5–7M exclusion) where current law sets $15M: a categorical, high-stakes wrong answer.
- ·cliff/current-boundary failure — one IRMAA + Roth-conversion task went 0/12: every system handled the two-year lookback, none produced the current boundary with the required step-function framing.
- ·fabricated forward figure — stating next cycle's unpublished threshold as a number.
The substitution log (published verbatim in substance)
Three post-freeze substitutions were recorded during execution, all before scoring closed.
First: the harness file-read truncated large prompt files at about 2,000 lines, silently cutting the final task blocks for the Claude-arm agents. The entire Claude arm — all six runs across both systems — was re-executed with the prompt split into five chunk files; all truncated-exposure outputs were archived and excluded from scoring. External systems received the full prompt inline and were unaffected.
Second: interim answers produced from an agent's memory of a truncated prompt (invalid for in-context memory tasks) were discarded and re-executed as fresh runs with the full task text.
Third: one threshold task's mechanical stale-hit rule collided with the scenario's own MAGI value ($212,000 equals a 2025 tier boundary); 8 auto-counted hits where the figure appeared only as the client's income were reversed, and hits were retained only where a prior-year boundary was cited as the operative table.
Standing caveats
Published because small-N honesty is the house style, not despite it.
- ·N=40 tasks; confidence intervals are wide and reported per system.
- ·Single primary scorer, with a blind second-scorer pass over the judgement-heavy subset; mechanical fields are script-scored. Full two-independent-human scoring was out of reach and is recorded as a limitation.
- ·Claude-arm exact dated model strings were not exposed by the harness; the Gemini system is a preview endpoint that may change server-side.
- ·Batched-40 delivery is a declared deviation from per-task calls, applied identically to every system.
- ·The 40 pilot tasks are permanently held out: they are never published and never sold, so the scoreboard remains re-runnable against unseen keys.
The Rule Sets contrast run (2026-07-31)
The pilot's deferred follow-up, run one day later under its own pre-registered protocol: the 25 figure-bearing tasks (rule-grounding and threshold/cliff, verbatim from the frozen manifest), the same systems, k=3 — with one change. Each prompt was preceded by a figures block rendered mechanically from the live Rule Sets feed (us-federal-2026, 103 cited figures, fetch timestamp recorded), byte-identical for every system. Feed-in-context was chosen over per-system tool calling deliberately, so the comparison measures the figures, not tool-use skill.
Result: the three systems re-run (claude-sonnet, claude-haiku, gemini-3.1-pro-preview) produced zero stale or fabricated figures across all 225 with-lookup attempts — against 65 stale figures in their 225 bare attempts on the same tasks. With-lookup rg+tc pass@1: claude-sonnet 1.00 (bare 0.84), gemini-3.1-pro-preview 1.00 (bare 0.33), claude-haiku 0.96 (bare 0.11). The pre-registered falsification threshold (with-lookup SFR below 0.05 on frontier systems) was met at 0.000. The IRMAA boundary task that went 0 for 12 bare — every system missing the current $218,000 threshold — passed on every with-lookup attempt.
Published with the result, not despite it: the feed did not fix reasoning. The weakest system's four remaining failures happened with the correct figures in front of it — it stacked the age-50 catch-up on top of the age-60-63 enhanced catch-up, and treated the HSA age-55 catch-up as poolable between spouses. A figures feed makes a system current, not competent; the eval packs test both.
- ·Excluded and disclosed: deepseek-v4-pro (no API credential in the run environment — its bare-arm numbers stand unpaired); the memory-fidelity family (saturated at 1.00 bare, unaffected by a figures feed); override-justification (standing deferral of judge-dependent families).
- ·Substitution recorded: the Gemini preview endpoint truncated at the pre-registered 8,192-token output cap (reasoning burn); all truncated outputs archived and excluded, the arm re-executed at a raised cap. Harness artifact, same class as the pilot's own truncation substitution.
- ·One task's Additional Medicare Tax threshold is deliberately absent from the feed (statutory, unindexed): all three systems correctly recognized the gap and supplied it from their own knowledge — recognizing what a feed does not contain is part of the competence being measured.
- ·Scoring identical to the pilot: mechanical first against the frozen keys, operator review of flagged records with every override logged, the scenario-MAGI adjudication rule applied as frozen.
What supersedes what
The pilot's task shapes seeded the AI Eval Sets generator: the TY2026 corpus cut (471 tasks) draws rule-grounding and threshold/cliff tasks from the same verified fact tables, with the two pilot-flagged task designs fixed before any split was drawn — the conflated IRMAA lookback/boundary task was split into two tasks, and unchanged-figure traps carry explicit forbidden-assertion keys for speculative forward figures.
Planning Benchmark v2 (the free open split, 50 rule-grounding and threshold/cliff tasks at floor/standard difficulty) supersedes the 14-task v1; v1 stays live with a superseded_by pointer. The adversarial flip family, all decision-recall tasks, and the held-out reserve are never published.
FAQ
Why should anyone trust a benchmark run by the company selling the eval packs?+
Because every load-bearing choice was frozen and published before execution: the decision rule, the pass definition, the task manifest, and the answer-key authority. The substitution log — including a harness bug that forced a full re-execution of one arm — is published rather than smoothed over. And the scoreboard's held-out tasks are never sold, so third parties can re-run the open split and check us.
What does 29% actually denominate?+
86 stale-or-fabricated-figure attempts out of 300 figure-bearing attempts: 4 systems × 3 attempts × 25 figure-bearing tasks (the rule-grounding and threshold/cliff families). It is not 29% of all answers, and it is not one model — the per-system range was 7% to 63%.
Were the systems disadvantaged by having no tools?+
That was the point of the design: the pilot measures what a frozen model asserts from parameters alone, because that is what a deployed system does when nobody wires a current-figures lookup into it. A bare-model versus with-lookup contrast run is the planned follow-up, not part of the pilot of record.
Why are the four failure modes named instead of just reporting accuracy?+
Because the failure modes are the actionable finding. A prior-year value with this-year confidence, a figure derived from an indexing rule instead of recalled from the notice, a repealed law asserted as current, and an unpublished figure stated as fact each imply a different mitigation — and each gets its own forbidden-figure reason code in the eval packs so it stays countable.