Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-5 outperforms federal judges in legal reasoning experiment

papers.ssrn.com

Post reply on HN
Print view View original

81–90 of 254 posts

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#81

Post by FarmerPotato »

Lerc wrote: post #71
scottLobster wrote: post #19

Earlier quoted context omitted.

Yeah, I'm reminded of the various child porn cases where the "perpetrator" is a stupid teenager who took nude pics of themselves and sent them to their boy/girlfriend. Many of those cases have been struck down by judges because the letter of the law creates a non-sequitur where the teenager is somehow a felon child predator who solely preyed on themselves, and sending them to jail and forcing them to sign up for a se…

This is one of the roles of justice, but it is also one of the reasons why wealthy people are convicted less often. While it often delivered as a narrative of wealth corrupting the system, the reality is that usually what they are buying is the justice that we all should have. So yes, a judge can let a stupid teenager off on charges of child porn selfies. but without the resources, they are more likely be told by a p…

Oddly enough, Texas passed reform to keep sexting teens from getting prosecuted when: they are both under 18 and less than two years difference in age. It was regarded as a model for other states. It's the only positive thing I have heard of Texas legislating wrt sexuality.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#82

Post by bcrosby95 »

qgin wrote: post #34

It seems that a lot of people would rather accept a relatively high risk of unfair judgement from a human than accept any nonzero risk of unfair judgement from a computer, even if the risk is smaller with the computer.

> even if the risk is smaller with the computer.

How do we even begin to establish that? This isn't a simple "more accidents" or "less accidents" question, its about the vague notion of "justice" which varies from person to person much less case to case.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#83

Post by TurdF3rguson »

gowld wrote: post #6
TurdF3rguson wrote: post #3

You can also avoid "hungry judge effect" by making sure GPT is always fully charged before prompting it.

"hungry judge effect" is a debunked myth.

It's been criticized, but "time since last meal" is still known to be a predictor of harsher sentences (even when you control for legal representation / severity).

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#84

Post by TurdF3rguson »

nkrisc wrote: post #47

Frankly I don’t care, I’ll take human judges any day, because they have something AI does not: flesh and bone and real skin in the game.

Real skin in the game is also known as bias. That's an example of something a judge should not have.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#85

Post by jfengel »

deepsun wrote: post #25
codingdave wrote: post #2

IANAL, but this seems like an odd test to me. Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls. If LLMs give only one answer, no matter what nuances are at play, that sounds like they are failing to judge and instead are diminishing the thought process down to black-…

The main job of a judicial system is to appear just to people. As long as people think it's just -- everyone is happy. But if it's strictly by the law, but people consider it's unjust -- revolutions happen. In both cases, lawmakers must adapt the law to reflect what people think is "just". That's why there are jury duty in some countries -- to involve people to the ruling, so they see it's just .

I've never met a lawyer who believes that. To a lawyer, justice requires agreement on the laws, rather than individual notions of justice. If the law is unjust, it's up to the lawmaking body to fix that. I hear this from lawyers of all ideologies.

I believe that this is absurd, but I'm not a lawyer.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#86

Post by Aurornis »

thewanderer1983 wrote: post #60

I was diagnosed with a rare blood disease called Essential Thrombocythemia (ET) which is part of a group of diseases called myeloproliferative neoplasms. This happened about three years ago. Recently, I decided to get a second opinion and my new specialist changed my diagnosis from ET to Polycythemia Vera (PV). She also highly recommended I quickly go and give blood to lower my haematocrit levels as it put me at a mu…

I’m glad you figured it out, but there are a lot of situations like this that look good with the benefit of hindsight.

I have some horror stories from a friend who started trusting ChatGPT over his doctors at the time and started declining rapidly. Be careful about accepting any one source as accurate.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#87

Post by bawolff »

codingdave wrote: post #2

IANAL, but this seems like an odd test to me. Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls. If LLMs give only one answer, no matter what nuances are at play, that sounds like they are failing to judge and instead are diminishing the thought process down to black-…

> Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls.

I disagree - law should be the same for everyone. Yes sometimes crimes have mitigating curcumstances and those should be taken into account. However that seems like a separate question of what is and is not illegal.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#88

Post by lukeinator42 »

overtone1000 wrote: post #27

I wonder whether the original study was in GPT-5's training data. I asked it whether this was the case, and it denied it, but I have no idea whether that result is credible.

I was also wondering this, and in one of the footnotes they say "Given that our experiment was conducted in 2025, one might wonder whether Kansas’ updated law is reflected in GPT’s training data and thus skews its decisions. We find no evidence of such contamination." when talking about a specific updated law. But how does one have 'no evidence of such contamination' without seeing the training data?

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#89

Post by seattle_spring »

ngetchell wrote: post #62

Count me out of a society that uses LLMs to make rulings. The dystopia of having to find a lawyer who is best at promoting the "unbiased" judge sounds like a hellscape.

Right? Especially considering the politics of some of the loudest AI evangelists. Do I want my fate decided by technology bankrolled by Peter Thiel, Elon Musk, Marc Andreessen, Mark Zuckerberg, or Jeff Bezos?

Hell no.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#90

Post by droidjj »

jmalicki wrote: post #56

The title is wrong. The title of the paper is "Silicon Formalism: Rules, Standards, and Judge AI" When they say legally correct they are clear that they mean in a surface formal reading of the law. They are using it to characterize the way judges vs. GPT-5 treat legal decisions, and leave it as an open question which is better. The conclusion of the paper is "Whatever may explain such behavior in judges and some LLMs…

> We find the LLM to be perfectly formalistic, applying the legally correct outcome in 100% of cases; this was significantly higher than judges, who followed the law a mere 52% of the time.
Post reply on HN