Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI outperforms law professors in Stanford Law study

law.stanford.edu

Post reply on HN
Print view View original

81–90 of 384 posts

Re: AI outperforms law professors in Stanford Law study

#81

Post by runarberg »

chewbacha wrote: post #40

My best guess is that Gemini was trained on the textbooks that the questions are meant to test against, thus they are probably better at explicit recall of those questions or related questions. This is a pretty limited introductory course based on what it says in the methods of the paper itself.

That and the research is done by Stanford’s HAI institute with an obvious bias and the paper is curiously missing a conflict of interest statement.

EDIT: just found out that Google is a major donor to HAI. So this research is at least partially funded by Google. Which is probably the reason the authors fail to declare no conflict of interest.

Re: AI outperforms law professors in Stanford Law study

#82

Post by thatcat »

zuzululu wrote: post #66
thewebguyd wrote: post #15

Earlier quoted context omitted.

I think this is probably true for most skilled professions. AI is best used in the hands of folks already knowledgeable in the skills/professions they are using it for. I liken it to me googling things as a sysadmin vs. Jane from accounting doing it. The non-tech end user is far more likely to make the problem worse, or install something sketchy from the ad riddled results than I am, or one of my help desk employees…

im not so sure i think devs overestimate their own role and underestimate others i am seeing lawyers and doctors roll out their own software with AI but we dont have their training and experience

So a software engineer could diagnose an illness with ai, even if they happen to be right that doesn't really prove much about how bad of an idea it could be in a long tail scenario.

Re: AI outperforms law professors in Stanford Law study

#83

Post by voxl »

wilg wrote: post #43
mitkebes wrote: post #37

Earlier quoted context omitted.

The arguments need to be based on actual law, and any cited reference cases need to be real. There's been a lot of news stories about lawyers using AI, and then getting in trouble for citing hallucinated laws or cases. It doesn't matter if the AI response is "preferred" over the human one if it gets thrown out when put under the scrutiny of a real case.

Who's gonna determine that? A bunch of law professors?

But did they? Or did they just go off what answer felt better? Did they put in any work to actually confirm the answer? Or did the busy law professors just click through and move on with their life?

Re: AI outperforms law professors in Stanford Law study

#84

Post by maxbond »

hoppyhoppy2 wrote: post #11
maxbond wrote: post #9

Earlier quoted context omitted.

Just so you know, I have nothing to do with Stanford, but I am flagging this as conspiratorial nonsense. So when you comment is flagged, I just want you to know that it doesn't confirm your belief, it's just that this comment harms discussion and so must be removed.

>Don't feed egregious comments by replying; flag them instead. If you flag, please don't also comment that you did. https://news.ycombinator.com/newsguidelines.html

Yes, mea culpa. Occasionally I break that rule on my own judgement. Feel free to flag my comment. (I think it's important to disconfirm conspiracy theories.)

Re: AI outperforms law professors in Stanford Law study

#85

Post by godelski »

thewebguyd wrote: post #15
causal wrote: post #7

As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.

I think this is probably true for most skilled professions. AI is best used in the hands of folks already knowledgeable in the skills/professions they are using it for. I liken it to me googling things as a sysadmin vs. Jane from accounting doing it. The non-tech end user is far more likely to make the problem worse, or install something sketchy from the ad riddled results than I am, or one of my help desk employees…

  > I think this is probably true for most skilled professions.
I agree, BUT I also find that it's easy for experts to atrophy quickly. When the AI is right 80/90% of the time it lulls you into over confidence.

I find those that are best and make the greatest use are the ones who remain skeptical but also use the tool. The same people who were already nuanced and picky before AI. The same people who already doubted and questioned their own work, and used that suspicion to help prevent them from having over confidence in their own work. If you weren't willing to just "lgtm" with your own code, it's difficult to do that with AI.

(To be clear, I'm not saying perfectionists. Some might call them that because the picky people have higher standards, but a good expert has to also understand that perfection doesn't exist. That's often a driving force in the suspicion! This also tends to cause them to continually improve)

Re: AI outperforms law professors in Stanford Law study

#86

Post by goodmythical »

rayiner wrote: post #22
causal wrote: post #7

As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.

I would think that LLMs would be better at avoiding foot-guns. That’s a situation where you have a list of well known rules and potential pit falls, and the work of the lawyer is to apply those to a fact pattern. That’s something that has been hard to automate programmatically, because the fact patterns are similar but different. LLMs, however, seem to excel at applying general principles to differing fact patterns.

I don't know the source off hand, but I've seen llms hallucinating case citations in order to "prove" their premises.

can't get more foot gun than "well according to [fiction] it is a well established practice (that the defendent is guilty)"

Re: AI outperforms law professors in Stanford Law study

#87

Post by atmavatar »

rayiner wrote: post #22
causal wrote: post #7

As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.

I would think that LLMs would be better at avoiding foot-guns. That’s a situation where you have a list of well known rules and potential pit falls, and the work of the lawyer is to apply those to a fact pattern. That’s something that has been hard to automate programmatically, because the fact patterns are similar but different. LLMs, however, seem to excel at applying general principles to differing fact patterns.

Instead, the LLMs create entirely new foot guns like citing non-existent cases. You can't go more than a week without encountering another news report of a lawyer submitting an AI-generated legal brief rife with bogus case citations, which even includes briefs submitted to state supreme courts.

e.g., https://www.npr.org/2026/04/03/nx-s1-5761454/penalties-stack...

Re: AI outperforms law professors in Stanford Law study

#88

Post by Aperocky »

> rated AI responses significantly higher than answers written by other professors, with AI winning 75% of head-to-head matchups.

That's the problem, you never know when the 25% deliver a true stink bomb, and that's not considering prompting - while a fair prompt/question maybe considered objective, it's very easy to stray.

Re: AI outperforms law professors in Stanford Law study

#89

Post by maxbond »

thin_carapace wrote: post #36
maxbond wrote: post #9

Earlier quoted context omitted.

Just so you know, I have nothing to do with Stanford, but I am flagging this as conspiratorial nonsense. So when you comment is flagged, I just want you to know that it doesn't confirm your belief, it's just that this comment harms discussion and so must be removed.

for what it's worth I have no idea why it would be nonsense to question institutional motivations especially in the context of an academic article that could easily be corporate propaganda, I also think that shutting conversations down is much more harmful than discussing topics that are potentially harmful

Completely unevidenced conspiracy theories can only harm the discussion. The only possible benefit is to disconfirm conspiracy theories and discourage paranoid thinking. The odds that Standford as an institution are astroturfing on HN round down to 0.

What they're almost certainly observing is that these critical comments are being flagged as inappropriate. People make inappropriate comments that happen to contain criticism all the time, and I frequently see people edit them to declare that they were flagged because the group they're criticizing is astroturfing. It's virtually never the case. I've never seen it happen.

But to be clear I am completely ambivalent on Stanford and if you want to criticize them, more power to you.

Re: AI outperforms law professors in Stanford Law study

#90

Post by KnuthIsGod »

In the hands of a domain expert, AI is useful. In the hands of the naive, it is a foot gun.

I killed my Arch installation and was stuck at the GRUB prompt.Unwilling to brush up my rusty knowledge of GRUB syntax, I asked Gemini for help. The commands Gemini suggested would have wiped my hd...

Once Gemini was told that I was using BTRFS, the suggestion from Gemini looked a bit more sane, but still looked incorrect to me.

It was only after I informed Gemini that I was using a NMVE with BTRFS that it finally produced a sane command.

Post reply on HN