When LLM judges agree, should we believe them?

Post by qarl » Mon, Sep 14, 2026, 10:26 PM UTC

emodendroket wrote: post #17

qarl wrote: post #10

While this is absolutely true - I'd hesitate to discount using similar agents for checking each other. Two agents will almost never hallucinate in the same way, regardless of their weights - and by having a second one (with a different context) check almost entirely eliminates the problem.

It depends what we're judging, doesn't it? If it's "is the formatting in this document compliant with our standards?" I think it's reasonable. If it's like, life-altering if it's wrong I'm less sanguine.

Post by knollimar » Tue, Sep 15, 2026, 12:43 AM UTC

piyh wrote: post #34

qarl wrote: post #10

While this is absolutely true - I'd hesitate to discount using similar agents for checking each other. Two agents will almost never hallucinate in the same way, regardless of their weights - and by having a second one (with a different context) check almost entirely eliminates the problem.

No modern LLM can tell how many eyes the magic card Pit Imp has. They all say 2. This is across all reasoning levels and paid Gemini, Claude, and GPT (Sol) Drawing a line red to split up the image then has them answer correctly. Their failure modes are highly correlated.

Post by someguynamedq » Tue, Sep 15, 2026, 2:39 AM UTC

qarl wrote: post #10

While this is absolutely true - I'd hesitate to discount using similar agents for checking each other. Two agents will almost never hallucinate in the same way, regardless of their weights - and by having a second one (with a different context) check almost entirely eliminates the problem.