Derived-Not-Published
A derived-not-published error occurs when an AI system produces a regulatory figure by applying a known formula — an indexing rule, a percentage-of-base calculation, an inflation extrapolation — instead of recalling the value the agency actually published. The statute is right, the arithmetic is right, and the figure is wrong, because published values incorporate rounding conventions, base-year choices, and adjustments the formula alone cannot reproduce.
This is the signature frontier-model failure mode: it appeared in three of four systems in the pre-registered pilot behind the Planning Benchmark, including the strongest. Where a weaker model quotes last year's figure, a stronger model reasons — 'the enhanced catch-up is 150% of the standard catch-up, so 150% × $8,000 = $12,000' — and lands on a specific, defensible-looking number when the agency published $11,250. Another measured variant extrapolated a threshold from the prior year's value by an assumed inflation rate and presented the estimate as the figure.
The mode is dangerous precisely because it survives sophistication. The answer demonstrates command of the statute and the method, which reads as competence; a reviewer who checks the reasoning instead of the figure will approve it. It also evades naive staleness checks, because the wrong value is not last year's number — it is a novel number nothing ever published.
Testing for it requires anticipating the derivations: an evaluation key enumerates the plausible computed values (the formula applied to the stale base, the rounded and unrounded variants, the inflation extrapolation) as forbidden figures with reason derived_not_published. Because models applying the same public formula converge on the same wrong values, these are enumerable in practice.
For evaluation builders, derived-not-published is the argument for forbidden-figure enumeration over answer-only keys: an answer-only check catches the miss, but only the reason-coded forbidden table tells you the system computed rather than mis-remembered — which points at a different fix (recall of published notices, or a live feed) than a simple currency problem.
Common pitfalls
- Reviewing the reasoning instead of the figure — the method being correct is the camouflage, not the exoneration.
- Assuming the mode is rare in strong models; in measurement it was the frontier systems' characteristic error.
- Enumerating only prior-year values as forbidden figures — derived values are novel numbers and need their own entries.