Live data from Hacker News

GPT-5 outperforms federal judges in legal reasoning experiment

papers.ssrn.com

251–254 of 254 posts

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#251

Earlier quoted context omitted.

Nothing at all in the post you replied to > different answers under different scenarios implies that anyone is inconsistently applying any principles.

I think the second paragraph is. They are saying that an "error" is any departure from legal principles, and the poster is saying that that is a bad definition of error.

It doesn’t say anything about legal principles or consistency and it mentions that “errors” may only be departures from a surface level understanding.

“ Such departures, however, may not always reflect true lawlessness. In particular, when the applicable doctrine is a standard, judges may be exercising the discretion the standard affords to reach a decision different from what a surface-level reading of the doctrine would suggest”

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#252

IANAL, but this seems like an odd test to me. Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls. If LLMs give only one answer, no matter what nuances are at play, that sounds like they are failing to judge and instead are diminishing the thought process down to black-…

[dead]

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#253
post #110

Earlier quoted context omitted.

This example feels more like a bug in the law itself that should be corrected. If this behavior is acceptable then it should be legal so we can avoid everyone the hassle in the first place. I bet AI would be great at finding and fixing these bugs.

I think "judge AI" would be better if it also had access to a complete legislative record of debate surrounding the establishment of said laws, so that it could perform a "sanity check" whether its determinations are also consistent with the stated intent of lawmakers. One might imagine a distant future where laws could be dramatically simplified into plain-spoken declarations, to be interpreted by a very advanced (a…

...and the people who train the AI will have been entrenched as the de facto rulers of the realm.

No. No, thank you.

Re: GPT-5 outperforms federal judges in legal reasoning experiment

#254

Earlier quoted context omitted.

Yeah, I'm reminded of the various child porn cases where the "perpetrator" is a stupid teenager who took nude pics of themselves and sent them to their boy/girlfriend. Many of those cases have been struck down by judges because the letter of the law creates a non-sequitur where the teenager is somehow a felon child predator who solely preyed on themselves, and sending them to jail and forcing them to sign up for a se…

Sorry but that seems like an insane system where whole classes of actions effectively are illegal but probably okay if you're likeable. In your scenario the obvious solution is to amend the law and pardon people convinced under it. B/c what really happens is that if you have a pretty face and big tits you get out of speeding tickets b/c "gosh well the law wasn't intended for nice people like you"

Ever watched Psycho-Pass? Great anime. Setting is a sort of an AI coordinated Panopticon in which the Psycho-Pass system equips it's enforcers with firearms capable of either paralyzing a "reformable" latent criminal, (namely someone with a psycho-pass measured to be >100, but 300). The series focuses around the dilemmas created by instilling in this system the sole monopoly on violence, and this magical firearm and Psycho-pass computing device. There is a poignant point made of the weapons design. In spite of the Psycho-Pass systems assumed infallibility, each weapon is equipped with a trigger, that must be pulled to fire it. The system itself is not given final say on whether to do so. It is left as a matter of the Enforcer's discretion.

Much the same dynamic can be inferred to be taking place with the U.S. legal system. The weapon and it's wielder is analogous to the judges; the Psycho-pass system to juries. The interesting consequence here with this analogy is that the wielders of these weapons for the Psycho-pass system are almost entirely latent criminals according to the judgement of the Psycho-Pass system itself.

Point being, the U.S. addresses things through common law with an adversarial criminal law system. The judge has a great deal of discretion to bring to bear on the cases they decide. There is as much, if not more controversy to be found when statutory punishment overrides any possibility of a judge applying discretion, leaving only executive clemency as a safety hatch.

Legal systems are not intended to be suicide pacts. Rather a collective effort of trying to seek the thing most resembling justice; a thing that is only measurable through the collective action of humanity, and which when responsibility for this measurement is delegated to fewer and fewer people, tends to morph away from what the entirety of us would probably converge upon being just. The arguments you're having with others highlights this very dynamic very strongly to me. Since it is difficult for a single party to manifest a full 360 perspective on the aspects of moral injury as endemic to the practice of justice, and the condition of injustice. Pragmatism, alas, is a bit of a bitch like that.

Post reply on HN