Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5 outperforms federal judges in legal reasoning experiment

papers.ssrn.com

Post reply on HN
Print view View original

101–110 of 254 posts

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#101

Post by NoahZuniga »

NoahZuniga wrote: post #93
bawolff wrote: post #87

Earlier quoted context omitted.

> Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls. I disagree - law should be the same for everyone. Yes sometimes crimes have mitigating curcumstances and those should be taken into account. However that seems like a separate question of what is and is not illegal.

The thing is, Laws do not forsee in all cases, and language is not completely objective, so you cannot avoid judgement calls. One example is computer hacking, which in many jurisdictions is specified in very vague terms.

Another example is that in the Netherlands, there's a crime called "valsheid in geschriften" which exists to make it easy to prosecute fraud. It states that if you create a document with false information with the intent to use that document to deceive, you can get up to 5 years of jail time or some really big fine. Is lying on a paper insurance form to get a cheaper premium breaking this law? This doesn't seem clear cut to me.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#102

Post by [deleted] »

bawolff wrote: post #87
codingdave wrote: post #2

IANAL, but this seems like an odd test to me. Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls. If LLMs give only one answer, no matter what nuances are at play, that sounds like they are failing to judge and instead are diminishing the thought process down to black-…

> Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls. I disagree - law should be the same for everyone. Yes sometimes crimes have mitigating curcumstances and those should be taken into account. However that seems like a separate question of what is and is not illegal.

[deleted]

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#103

Post by wahern »

jfengel wrote: post #85
deepsun wrote: post #25

Earlier quoted context omitted.

The main job of a judicial system is to appear just to people. As long as people think it's just -- everyone is happy. But if it's strictly by the law, but people consider it's unjust -- revolutions happen. In both cases, lawmakers must adapt the law to reflect what people think is "just". That's why there are jury duty in some countries -- to involve people to the ruling, so they see it's just .

I've never met a lawyer who believes that. To a lawyer, justice requires agreement on the laws, rather than individual notions of justice. If the law is unjust, it's up to the lawmaking body to fix that. I hear this from lawyers of all ideologies. I believe that this is absurd, but I'm not a lawyer.

In Federal courts mandatory minimum sentences were judged to be unconstitutional, as the ability to individualize sentencing was considered a prerogative intrinsic to the role of [Federal] judges. Though, a judge cannot impose a sentence greater than the maximum allowed under law. Federal courts still have sentencing guidelines that are almost always applied, but strictly speaking they're advisory.

More fundamentally, individualized justice is a core principle of common law courts, at least historically speaking. It's also an obscure principle, but you can't fully understand the system without it, including the wide latitude judges often wield in various (albeit usually highly technical) aspects of their job.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#104

Post by cindyllm »

themafia wrote: post #35

> In fact, the LLM makes no errors at all. hah. Sure. > Subjects were told that they were a judge who sat in a certain jurisdiction (either Wyoming or South Dakota), and asked to apply the forum state’s choice of law rule to determine whether Kansas or Nebraska law should apply to a tort case involving an automobile accident that took place in either Kansas or Nebraska. Oh. So it "made no errors at all" with respect…

[dead]

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#105

Post by snitty »

codingdave wrote: post #2

IANAL, but this seems like an odd test to me. Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls. If LLMs give only one answer, no matter what nuances are at play, that sounds like they are failing to judge and instead are diminishing the thought process down to black-…

So here the test was effectively given a set of relevant facts, can we influence the way a judge (or LLM) rules based on superfluous facts. The judges were either confused or swayed by the superfluous facts. The LLM was not. The matter was one where the outcome should have been determinative, not judgment-based, under US law.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#106

Post by contrarian1234 »

scottLobster wrote: post #19
codingdave wrote: post #2

IANAL, but this seems like an odd test to me. Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls. If LLMs give only one answer, no matter what nuances are at play, that sounds like they are failing to judge and instead are diminishing the thought process down to black-…

Yeah, I'm reminded of the various child porn cases where the "perpetrator" is a stupid teenager who took nude pics of themselves and sent them to their boy/girlfriend. Many of those cases have been struck down by judges because the letter of the law creates a non-sequitur where the teenager is somehow a felon child predator who solely preyed on themselves, and sending them to jail and forcing them to sign up for a se…

Sorry but that seems like an insane system where whole classes of actions effectively are illegal but probably okay if you're likeable. In your scenario the obvious solution is to amend the law and pardon people convinced under it. B/c what really happens is that if you have a pretty face and big tits you get out of speeding tickets b/c "gosh well the law wasn't intended for nice people like you"

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#107

Post by [deleted] »

NoahZuniga wrote: post #101
NoahZuniga wrote: post #93

Earlier quoted context omitted.

The thing is, Laws do not forsee in all cases, and language is not completely objective, so you cannot avoid judgement calls. One example is computer hacking, which in many jurisdictions is specified in very vague terms.

Another example is that in the Netherlands, there's a crime called "valsheid in geschriften" which exists to make it easy to prosecute fraud. It states that if you create a document with false information with the intent to use that document to deceive, you can get up to 5 years of jail time or some really big fine. Is lying on a paper insurance form to get a cheaper premium breaking this law? This doesn't seem clear…

[deleted]

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#108

Post by treis »

nkrisc wrote: post #47

Frankly I don’t care, I’ll take human judges any day, because they have something AI does not: flesh and bone and real skin in the game.

Not really. Ultimately it's just a job and a job without any tangible benefit to doing well.

Most regular folk that end up in front of a judge would do well to have a quick and predictable decision. It's months to years before things happen in court and are usually gated behind 10s of thousands in legal fees or a ton of effort. To have a judge bot available for a decision immediately is enormously beneficial.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#109

Post by bdangubic »

treis wrote: post #108
nkrisc wrote: post #47

Frankly I don’t care, I’ll take human judges any day, because they have something AI does not: flesh and bone and real skin in the game.

Not really. Ultimately it's just a job and a job without any tangible benefit to doing well. Most regular folk that end up in front of a judge would do well to have a quick and predictable decision. It's months to years before things happen in court and are usually gated behind 10s of thousands in legal fees or a ton of effort. To have a judge bot available for a decision immediately is enormously beneficial.

> … predictable decision

can’t have this from a system which is by its nature non-deterministic

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#110

Post by a13n »

scottLobster wrote: post #19
codingdave wrote: post #2

IANAL, but this seems like an odd test to me. Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls. If LLMs give only one answer, no matter what nuances are at play, that sounds like they are failing to judge and instead are diminishing the thought process down to black-…

Yeah, I'm reminded of the various child porn cases where the "perpetrator" is a stupid teenager who took nude pics of themselves and sent them to their boy/girlfriend. Many of those cases have been struck down by judges because the letter of the law creates a non-sequitur where the teenager is somehow a felon child predator who solely preyed on themselves, and sending them to jail and forcing them to sign up for a se…

This example feels more like a bug in the law itself that should be corrected. If this behavior is acceptable then it should be legal so we can avoid everyone the hassle in the first place. I bet AI would be great at finding and fixing these bugs.
Post reply on HN