Verify, Then Vote

The approach used Claude Sonnet 5.5 as a separate checker that saw only an answer sentence and a cited passage, judging support without access to the correct answer, and the team formally verified in Lean 4 that any swap from a correct to a wrong answer must trace to a source check backing the wrong answer. On the 526 questions where models split between right and wrong, the rule fixed 91 vote errors while breaking 12 correct answers, lifting accuracy from 71.5% to 86.5%, though part of the gain stemmed from deterministic tie-breaking versus the vote's random ties. The result matters because model voting cannot catch correlated errors, and source-checked selection offers a partial safeguard with an audit trail. The main caveat: of 376 answers the vote got right, the rule broke 12 (3.2%), and allowing for chance the true rate could reach 5.5%, exceeding the team's preset 5% limit—so the method failed its own guardrail. Watch whether larger or more diverse model pools and stricter checker prompts bring the breakage rate under the threshold.