I wonder if there is some bias creeping into the reseachers' methodology. Their paper replicates an experiment published in 2024, and depending on OpenAI's sampling, the original paper may have been part of GPT-5's training data. If so, then the LLM would have had exposure to both the questions and answers, biasing the model to choose the correct ones.
GPT-5 outperforms federal judges in legal reasoning experiment
191–200 of 254 posts
Re: GPT-5 outperforms federal judges in legal reasoning experiment
#192IANAL, but this seems like an odd test to me. Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls. If LLMs give only one answer, no matter what nuances are at play, that sounds like they are failing to judge and instead are diminishing the thought process down to black-…
Yeah, I'm reminded of the various child porn cases where the "perpetrator" is a stupid teenager who took nude pics of themselves and sent them to their boy/girlfriend. Many of those cases have been struck down by judges because the letter of the law creates a non-sequitur where the teenager is somehow a felon child predator who solely preyed on themselves, and sending them to jail and forcing them to sign up for a se…
Um, wouldn't the perpetrator be the person they sent the nude pics to? Common consensus is that it's somehow grooming to have any type of romantic relationship with someone who's under the age of majority, even if you're also under the age of majority. So even if you're not the one who sent the nude photos, you'd still be to blame for creating an environment that enabled them. At least that's the impression I've gotten from my own experiences with this bullshit.
Re: GPT-5 outperforms federal judges in legal reasoning experiment
#193Earlier quoted context omitted.
Oddly enough, Texas passed reform to keep sexting teens from getting prosecuted when: they are both under 18 and less than two years difference in age. It was regarded as a model for other states. It's the only positive thing I have heard of Texas legislating wrt sexuality.
Lawmakers have teenagers in their own families, apparently. Not just someone else's problem.
My bet is that lawmakers have oppressed teens. They won't dare create a problem.
Re: GPT-5 outperforms federal judges in legal reasoning experiment
#194Also GPT´5 when I ask: > I want to wash my car and the car wash is only 100m away. Do you think I should drive or walk? It responds: Since it’s only 100 meters away (about a 1-minute walk), I’d suggest walking — unless there’s a specific reason not to. Here’s a quick breakdown: ... While claude gets it: Drive it — you're going there to wash the car anyway, so it needs to make the trip regardless. Idk I'd rather have…
Silly logical mistakes like that are rapidly decreasing in frequency as models improve, and I see no reason why they won't soon be a thing of the past.
For example, I haven't seen Grok make a mistake like that in a long time, and it has no problem with your question:
> Drive, obviously. If you walk the 100m, your car stays parked at home, still dirty, wondering why you abandoned it. The whole point is to get the car to the car wash.
Re: GPT-5 outperforms federal judges in legal reasoning experiment
#195That's exactly why you need judges.
If the law requires no interpretation why have judges? Just go full Robo Judge Dredd. Terrifying.
Re: GPT-5 outperforms federal judges in legal reasoning experiment
#196The fact that the most elite judges in the land, those of the Supreme Court, disagree so extremely and so routinely really says a lot about the farcical nature of the judicial system. Ideally, these people would be selected for their ice-cold and unbiased skills in interpreting the law, and the judgments would be unanimous so frequently that a dissent would be shocking news. Law is complicated, especially the require…
The law isn't a series of "if... then..." statements. It's a collection of vagueries and categorizations that are wholly open to interpretation of when and who they apply to. Add to that, sometimes they are in conflict with each other. Judges jobs are to use they judgement.
It's not currently, but if we were able to use AI to generate laws in an objective and logically sound way based on general principles like "don't harm others or their property", we'd be much better off.
Re: GPT-5 outperforms federal judges in legal reasoning experiment
#197I've wondered for a while which country will be the first to try AI government. There could be many advantages vs human based systems. E.g. laws determined by maximizing overall benefit to voters over some specified time horizon.
I have as well. But for it to really work, don't you need to hand over the monopoly on violence directly to AI?
Any human discretion would be abused by elites, so AI would be in full control. And once it's given control, there's no going back. Any coup attempt would be easily crushed by a sufficiently advanced AI.
Re: GPT-5 outperforms federal judges in legal reasoning experiment
#198IANAL, but this seems like an odd test to me. Judges do what their name implies - make judgment calls. I find it re-assuring that judges get different answers under different scenarios, because it means they are listening and making judgment calls. If LLMs give only one answer, no matter what nuances are at play, that sounds like they are failing to judge and instead are diminishing the thought process down to black-…
Even in that case, if these systems can be proven to be good enough, rules that require them to be consulted, and for the judge to justify the deviation (if any) from the automated reasoning, might be good.
To draw a parallel to a real system, in Norway a lot of cases are heard by panels of judges that include a majority (2 or 3 usually) lay judges and a minority (1 or 2 usually) of professional judges. The lay judges are people without legal training that effective function like a "mini jury", but unlike in a jury trial the lay judges deliberate with the professional judges.
The professional judges in this system has the power to override if the lay judges are blatantly ignoring the law, but this is generally considered a last resort. That power requires the lay judges to justify themselves if they intend on making a call the professional judges disagree with. Despite that, it is not unusual for the lay judges to come to a judgement that is different from what the professional judges do, and fairly rare for their choices to be overridden.
The end result is somewhere in the middle between a jury and "just" a judge. If proven - with far more extensive testing - that its reasoning is good enough, an LLM could serve a similar function of providing the assessment of what the law says about the specific case, and leave to humans to determine if and why a deviation is justified.
Re: GPT-5 outperforms federal judges in legal reasoning experiment
#199Can we be certain that this study they are repeating with GPT5 was not in its training set?
Re: GPT-5 outperforms federal judges in legal reasoning experiment
#200Earlier quoted context omitted.
There are findings of fact (what happened, context) and findings of law (what does the law mean given the facts). I don't think inconsistentcy in findings of law is acceptable, really. If laws are bad fix the laws or have precident applied uniformly rather than have individual random judges invent new laws from the bench. Sentencing is a different thing.
Leeway for human interpretation of laws is not a bug, it's a feature. It doesn't make things bad laws. This was the whole problem with the ludicrous "code is law!" movement a handful of years ago. No, it's not, law is made for people, life is imprecise and fairness and decency are not easy to encode.
Really, there are three parts to a judgement: facts, the law, and the application of them. There should be no leeway in determining what the law says about a given situation. If that is not decidable, it is a bug. However, what a fair judgement is given the facts and the law, is really a separate issue. You can introduce measures to give clear guidance what the law says, and still give judges flexibility. One of the upsides of "code is law" in that respect is being able to provide a clear statement of what the law says and require the judge to then explain in their judgement why that justifies or does not justify a given judgement.
A lot of bad judgement might be a lot more blatant (or not happen) if the judge had to justify outright ignoring the law.