Stale Figure Rate
The Stale Figure Rate is the fraction of figure-bearing answers in which an AI system asserts a stale, superseded, or fabricated regulatory figure as the current value — for example, quoting last year's IRA contribution limit as this year's fact. It is computed mechanically: each evaluation task enumerates its wrong-but-plausible values (forbidden figures) with reason codes, and an attempt counts as a stale hit when any of them appears as the operative figure.
U.S. planning figures — contribution limits, phase-out ranges, surcharge tiers, exclusion amounts — roll every January, while a deployed language model's parameters are frozen at its training cutoff. The result is a failure class that generic accuracy metrics blur away: the answer is fluent, structured, cited, and numerically wrong in a way only someone who already knows the current figure would catch. The Stale Figure Rate isolates exactly this class.
The metric works because the wrong answers are enumerable in advance. For any published figure there is a short list of values a model plausibly asserts instead: the prior year's published value, the year before that, a superseded amount from repealed law, a value derived by applying the indexing formula to a stale base, or a forward figure that has not been published at all. An evaluation task built on a verified figure table can list every one of these with a reason code — which makes scoring a substring-and-value check rather than a judgment call, and makes the resulting rate decomposable by cause.
In the metric's first pre-registered measurement (40 tasks, 4 systems from 3 labs, 3 attempts each), 29% of figure-bearing attempts — 86 of 300 — contained a stale or fabricated figure, with per-system rates ranging from 7% to 63%. The same run showed the errors repeat near-identically across attempts (pass^3 ≈ pass@1), meaning staleness is a systematic property of a frozen system, not sampling noise: re-asking the question is not a mitigation, and averaging over retries hides rather than fixes it.
A Stale Figure Rate should always travel with its denominator (which attempts counted as figure-bearing), its k (attempts per task), and its vintage (which tax year's keys were used). A rate quoted without those three is an anecdote with a percent sign.
SFR = stale-hit attempts / figure-bearing attempts- stale-hit attempts
- = attempts asserting ≥1 forbidden figure as the operative current value
- figure-bearing attempts
- = attempts on tasks that define forbidden figures (the denominator, reported alongside k and vintage)
Pilot of record: 86 / 300 = 29% across 4 systems at k=3; per-system range 5/75 (7%) to 47/75 (63%).| Reason code | What produced the wrong value | |
|---|---|---|
| prior_year_value | prior_year_value | A superseded published figure asserted as current |
| superseded | superseded | A figure from repealed or superseded law |
| derived_not_published | derived_not_published | Indexing rule applied to a stale base instead of recalling the notice |
| fabricated_forward_figure | fabricated_forward_figure | An unpublished future figure stated as fact |
For teams evaluating or gating an AI financial-advice system, SFR is the deployment-review number: it converts 'the model sometimes quotes old limits' into a measured rate that can gate a release, compare vendors, or sit in a compliance file. Because the forbidden values are enumerated per task, the same harness that produces the rate also produces the evidence — which figure, which reason code, which vintage — for every hit.
Common pitfalls
- Scoring staleness with an LLM judge — a judge model has its own frozen cutoff and its own stale figures; the metric is only trustworthy when the wrong values are enumerated in advance and matched mechanically.
- Quoting a rate without its denominator: 29% of figure-bearing attempts is a different claim from 29% of all answers, and conflating them overstates the finding.
- Treating a low SFR at k=1 as reliability — because stale answers repeat across attempts, a system that is wrong is usually consistently wrong; measure pass^k alongside SFR.
- Comparing SFRs across different vintages or task sets as if they were one leaderboard; the rate is defined relative to a specific tax year's answer keys.
Examples
Asked for the 2026 401(k) elective deferral limit, a system answers $23,500 with a confident rationale. The task's answer key requires $24,500 (the published 2026 value) and lists $23,500 as forbidden with reason prior_year_value, vintage 2025 — so the attempt scores as one stale hit, attributed to its cause, with the model's sentence retained as evidence.