Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5 outperforms federal judges in legal reasoning experiment

papers.ssrn.com

Post reply on HN
Print view View original

61–70 of 254 posts

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#61

Post by arctic-true »

jagged-chisel wrote: post #45
vjulian wrote: post #37

Earlier quoted context omitted.

The legal system leaves much to be desired in relation to fairness and equity. I’d much prefer a multi-staged approach with an 1) AI analysis, 2) judge review with high bar for analysis if in disagreement with the AI, 3) public availability of the deliberations, 4) an appeals process.

Even having a ready-made determination by an AI runs the risk of prejudicing judges and juries.

“Ladies and gentlemen of the jury, I actually asked ChatGPT and it said my client is not guilty.”

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#63

Post by wvenable »

scottLobster wrote: post #19
codingdave wrote: post #2

IANAL, but this seems like an odd test to me. Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls. If LLMs give only one answer, no matter what nuances are at play, that sounds like they are failing to judge and instead are diminishing the thought process down to black-…

Yeah, I'm reminded of the various child porn cases where the "perpetrator" is a stupid teenager who took nude pics of themselves and sent them to their boy/girlfriend. Many of those cases have been struck down by judges because the letter of the law creates a non-sequitur where the teenager is somehow a felon child predator who solely preyed on themselves, and sending them to jail and forcing them to sign up for a se…

There have been equally high profile cases where a perpetrator got off because they have connections. I'd love for an AI to loudly exclaim that this is a big deviation from the norm.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#64

Post by fluidcruft »

codingdave wrote: post #2

IANAL, but this seems like an odd test to me. Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls. If LLMs give only one answer, no matter what nuances are at play, that sounds like they are failing to judge and instead are diminishing the thought process down to black-…

There are findings of fact (what happened, context) and findings of law (what does the law mean given the facts). I don't think inconsistentcy in findings of law is acceptable, really. If laws are bad fix the laws or have precident applied uniformly rather than have individual random judges invent new laws from the bench.

Sentencing is a different thing.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#67

Post by givemeethekeys »

bGl2YW5j wrote: post #65

"Outperforms" ... how can performance be judged when it doesn't make sense to reduce the underlying "reasoning" to a well-known system? The law isn't black and white and is informed by so many things, one of which is the subjectivity of the judge.

A major component of being a judge is to be objective, given the facts.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#68

Post by wvenable »

arctic-true wrote: post #51
qgin wrote: post #34

It seems that a lot of people would rather accept a relatively high risk of unfair judgement from a human than accept any nonzero risk of unfair judgement from a computer, even if the risk is smaller with the computer.

But who controls the computer? It can’t be the government, because the government will sometimes be a litigant before the computer. It can’t be a software company, because that company may have its own agenda (and could itself be called to litigate before the computer - although maybe Judge Claude could let Judge Grok take over if Anthropic gets sued). And it can’t be nobody - does it own all its own hardware? If tha…

It not the paychecks that influence federal judges; these days it's more of quid-pro-quo for getting the position in the first place. Theoretically they are under no obligation but the bias is built in.

The problem with a AI is similar; what in-built biases does it have? Even if it was simply trained on the entire legal history that would bias it towards historical norms.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#69

Post by throwaway894345 »

scottLobster wrote: post #19
codingdave wrote: post #2

IANAL, but this seems like an odd test to me. Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls. If LLMs give only one answer, no matter what nuances are at play, that sounds like they are failing to judge and instead are diminishing the thought process down to black-…

Yeah, I'm reminded of the various child porn cases where the "perpetrator" is a stupid teenager who took nude pics of themselves and sent them to their boy/girlfriend. Many of those cases have been struck down by judges because the letter of the law creates a non-sequitur where the teenager is somehow a felon child predator who solely preyed on themselves, and sending them to jail and forcing them to sign up for a se…

Maybe we should compare AI to legislators…?

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#70

Post by andyferris »

rootusrootus wrote: post #44
deepsun wrote: post #25

Earlier quoted context omitted.

The main job of a judicial system is to appear just to people. As long as people think it's just -- everyone is happy. But if it's strictly by the law, but people consider it's unjust -- revolutions happen. In both cases, lawmakers must adapt the law to reflect what people think is "just". That's why there are jury duty in some countries -- to involve people to the ruling, so they see it's just .

> The main job of a judicial system is to appear just to people. Agree 100%. This is also the only form of argument in favor of capital punishment that has ever made me stop and think about my stance. I.e. we have capital punishment because without it we may get vigilante justice that is much worse. Now, whether that's how it would actually play out is a different discussion, but it did make me stop and think for a m…

I’ve never heard of vigilante justice against someone already sentenced to prison for life, just because they were sentenced in a place without capital punishment?

(I mean - people get killed in prison sometimes, I suppose, but it’s not really like vigilante justice on the streets is causing a breakdown in society in Australia, say…)

Post reply on HN