Validating LLM Answers Without Ground Truth: Cross-Model Disagreement as a Health Signal

The create/validate pattern

The method is deliberately simple: whichever model family generated a system's answers, a different family audits them. The validator sees exactly what the answering system saw — the question and the memory content it actually retrieved — plus the proposed answer, and judges whether that answer is supported by that retrieved content. It never sees the ground-truth answer key, and it never modifies the answer being checked; its only output is a verdict and a disagreement report published alongside the score.

The distinction the validator is asked to make is narrow and specific on purpose: judge whether the retrieved content supports the answer, not whether the answer is true in the real world. A validator has no independent access to the underlying record, so 'true' isn't even a question it could answer reliably — 'supported by what was actually retrieved' is the question it's positioned to answer well, and if the retrieved content doesn't contain the information and the system guessed anyway, that counts as unsupported.

// Illustrative — the real mechanism ships as bench/validate.py
function auditAnswer(question, retrievedContext, proposedAnswer) {
  // The validator model NEVER receives the ground-truth answer key.
  const verdict = judgeModel.evaluate({
    question,
    context: retrievedContext,   // exactly what the answering system saw
    answer: proposedAnswer,
    instruction:
      "Judge ONLY whether this answer is supported by the context " +
      "above. Judge support, not real-world truth — you have no " +
      "access to the underlying records.",
  });
  return { supported: verdict.supported, reason: verdict.reason };
}

Why the family has to be different

The audit only means something if the validator's judgment is genuinely independent of whatever produced the answer. A model auditing its own family's output risks sharing the same blind spots that produced the answer in the first place — a systematic misreading that led the answering model astray could just as easily lead the same family's judge to the same misreading. Cross-family validation (one provider's model creates the answer, a different provider's model audits it) is what makes the check worth running: the two model families arrived at their behavior through different training, so agreement between them is closer to real corroboration, and disagreement is closer to a real signal rather than a coin flip.

Does it actually track quality? Checking against real ground truth

The method is only useful if its disagreement rate actually correlates with real correctness — otherwise it's just a second opinion with no calibration behind it. DecisionSynth Bench checked this directly by running the cross-validator on systems where the real exact-match score, against genuine ground truth, was also known. The disagreement rates ordered the systems the same way the real exact-match scores did:

Running it on your own system

The practical setup mirrors the create/validate framing directly: pick two genuinely independent model families (DecisionSynth Bench's own runs use Gemini and Claude in each direction), have one produce your system's answers along with the retrieved context behind each one, and have the other audit every answer against only that retrieved context — never against a ground-truth key, because in the scenario this method is built for, one doesn't exist.

Where this fits relative to real ground-truth scoring

Cross-model disagreement is a monitoring and triage tool, not a replacement for scoring against genuine ground truth wherever ground truth is available. Held-out evaluation with a real answer key remains the standard for a published benchmark score, precisely because it checks something the validator structurally cannot: whether the answer is actually correct, not just whether it's supported by whatever the system happened to retrieve. The validator's real value shows up specifically in the gap real ground-truth scoring can't reach — ongoing production monitoring, private-corpus health checks, or any setting where withholding the answer key is the whole point.

Key takeaways

• Cross-model disagreement lets a different model family audit an answer against exactly the retrieved content the answering system saw, without ever seeing a ground-truth key.
• The validator is explicitly instructed to judge support (is this answer backed by what was retrieved), not real-world truth — it has no independent access to the underlying record to judge truth against.
• The two model families must be genuinely independent — a model auditing its own family's output risks sharing the same blind spots that produced the answer.
• Disagreement rates have been shown to order systems the same way real exact-match scores do, from lowest disagreement on the strongest real performer to highest disagreement on the weakest — evidence the method tracks quality, not just noise.
• This is a monitoring and triage tool for settings without ground truth, not a substitute for held-out scoring against a real answer key wherever one is available.

FAQ