Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI outperforms law professors in Stanford Law study

law.stanford.edu

Post reply on HN
Print view View original

31–40 of 384 posts

Re: AI outperforms law professors in Stanford Law study

#31

Post by gamerDude »

mylifeandtimes wrote: post #21
wilg wrote: post #13

Earlier quoted context omitted.

What criteria would you use for judging legal arguments?

maybe seeing if the case law it cited was real or imagined? Just one idea, IANAL

Well, they had the data around if the answer would be harmful to the students learning. AI was scored at 3.5% harmful answers and 12% of law professor answers were considered harmful.

Re: AI outperforms law professors in Stanford Law study

#32

Post by Boss0565 »

causal wrote: post #7

As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.

cannot IMAGINE letting ai write my will rn.

Re: AI outperforms law professors in Stanford Law study

#33

Post by knollimar »

causal wrote: post #7

As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.

I'm afraid since claude cheats in benches, what will it do with law?

Re: AI outperforms law professors in Stanford Law study

#35

Post by applfanboysbgon »

Esophagus4 wrote: post #27
applfanboysbgon wrote: post #20

Earlier quoted context omitted.

Bad legal advice will keep you dealing with the legal system for much longer and at much greater cost. Something being cheap and quick upfront doesn't mean it will be cheap and quick by the end of the process.

But isn’t this study saying that the legal advice could actually be better with AI? A bit of extrapolation from the study, but not a crazy stretch.

Maybe, although I would be extremely hesitant to extrapolate from this one study and trust my legal life to an LLM. One thing that's worth noting, though, is that regardless of the quality of objective legal advice in the abstract, for a lot of smaller scale stuff the human connection actually is literally what is important. There are ambiguities in the law, which are not resolved deterministically but rather at the individual discretion of judges. Your lawyer, if they're any good at their job, knows the local judges and how they're likely to rule for given circumstances, which can influence their legal advice to you specifically.

Re: AI outperforms law professors in Stanford Law study

#36

Post by thin_carapace »

maxbond wrote: post #9
fgh_ask wrote: post #5

[flagged]

Just so you know, I have nothing to do with Stanford, but I am flagging this as conspiratorial nonsense. So when you comment is flagged, I just want you to know that it doesn't confirm your belief, it's just that this comment harms discussion and so must be removed.

for what it's worth I have no idea why it would be nonsense to question institutional motivations especially in the context of an academic article that could easily be corporate propaganda, I also think that shutting conversations down is much more harmful than discussing topics that are potentially harmful

Re: AI outperforms law professors in Stanford Law study

#37

Post by mitkebes »

wilg wrote: post #13
jshier wrote: post #10

Earlier quoted context omitted.

I do wish they'd used some more objective criteria. Simply being preferable one of the things LLMs have trained for since the beginning, hence its sycophantic nature.

What criteria would you use for judging legal arguments?

The arguments need to be based on actual law, and any cited reference cases need to be real.

There's been a lot of news stories about lawyers using AI, and then getting in trouble for citing hallucinated laws or cases. It doesn't matter if the AI response is "preferred" over the human one if it gets thrown out when put under the scrutiny of a real case.

Re: AI outperforms law professors in Stanford Law study

#39

Post by 12AHg »

bko wrote: post #30

Marc Andreessen argued that we've already reached AGI. He says that the top AI models give better answers than 99% of people he has access to, and he has access to some of the best people in their field. I'm getting more convinced. I mean, sure it makes dumb mistakes sometimes but its a particular set of self serving mistakes, commenting out tests in order to pass. We obv don't want this behavior but I wouldn't say i…

[flagged]

Re: AI outperforms law professors in Stanford Law study

#40

Post by chewbacha »

My best guess is that Gemini was trained on the textbooks that the questions are meant to test against, thus they are probably better at explicit recall of those questions or related questions.

This is a pretty limited introductory course based on what it says in the methods of the paper itself.

Post reply on HN