GPT-5 outperforms federal judges in legal reasoning experiment

Post by mullingitover » Thu, Feb 12, 2026, 4:44 AM UTC

parineum wrote: post #128

mullingitover wrote: post #127

The fact that the most elite judges in the land, those of the Supreme Court, disagree so extremely and so routinely really says a lot about the farcical nature of the judicial system. Ideally, these people would be selected for their ice-cold and unbiased skills in interpreting the law, and the judgments would be unanimous so frequently that a dissent would be shocking news. Law is complicated, especially the require…

The law isn't a series of "if... then..." statements. It's a collection of vagueries and categorizations that are wholly open to interpretation of when and who they apply to. Add to that, sometimes they are in conflict with each other. Judges jobs are to use they judgement.

Post by thaumasiotes » Thu, Feb 12, 2026, 5:01 AM UTC

NoahZuniga wrote: post #101

NoahZuniga wrote: post #93

Earlier quoted context omitted.

The thing is, Laws do not forsee in all cases, and language is not completely objective, so you cannot avoid judgement calls. One example is computer hacking, which in many jurisdictions is specified in very vague terms.

Another example is that in the Netherlands, there's a crime called "valsheid in geschriften" which exists to make it easy to prosecute fraud. It states that if you create a document with false information with the intent to use that document to deceive, you can get up to 5 years of jail time or some really big fine. Is lying on a paper insurance form to get a cheaper premium breaking this law? This doesn't seem clear…

Post by rgoldfinger » Thu, Feb 12, 2026, 5:09 AM UTC

swisniewski wrote: post #14

The premise seems flawed. From the paper: “we find that the LLM adheres to the legally correct outcome significantly more often than human judges” That presupposes that a “legally correct” outcome exists The Common Law, which is the foundation of federal law and the law of 49/50 states, is a “bottom up” legal system. Legal principals flow from the specific to the general. That is, judges decided specific cases based…

Post by latchkey » Thu, Feb 12, 2026, 5:14 AM UTC

scottLobster wrote: post #19

codingdave wrote: post #2

IANAL, but this seems like an odd test to me. Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls. If LLMs give only one answer, no matter what nuances are at play, that sounds like they are failing to judge and instead are diminishing the thought process down to black-…

Yeah, I'm reminded of the various child porn cases where the "perpetrator" is a stupid teenager who took nude pics of themselves and sent them to their boy/girlfriend. Many of those cases have been struck down by judges because the letter of the law creates a non-sequitur where the teenager is somehow a felon child predator who solely preyed on themselves, and sending them to jail and forcing them to sign up for a se…

Post by Auracle » Thu, Feb 12, 2026, 5:26 AM UTC

a13n wrote: post #110

scottLobster wrote: post #19

Earlier quoted context omitted.

Yeah, I'm reminded of the various child porn cases where the "perpetrator" is a stupid teenager who took nude pics of themselves and sent them to their boy/girlfriend. Many of those cases have been struck down by judges because the letter of the law creates a non-sequitur where the teenager is somehow a felon child predator who solely preyed on themselves, and sending them to jail and forcing them to sign up for a se…

This example feels more like a bug in the law itself that should be corrected. If this behavior is acceptable then it should be legal so we can avoid everyone the hassle in the first place. I bet AI would be great at finding and fixing these bugs.