Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI outperforms law professors in Stanford Law study

law.stanford.edu

Post reply on HN
Print view View original

71–80 of 384 posts

Re: AI outperforms law professors in Stanford Law study

#71

Post by HappMacDonald »

rayiner wrote: post #22
causal wrote: post #7

As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.

I would think that LLMs would be better at avoiding foot-guns. That’s a situation where you have a list of well known rules and potential pit falls, and the work of the lawyer is to apply those to a fact pattern. That’s something that has been hard to automate programmatically, because the fact patterns are similar but different. LLMs, however, seem to excel at applying general principles to differing fact patterns.

I would categorize this in the "expertise that people internalize but never figure out how to verbalize" department, and that is a department we have no way to teach an LLM because if nobody is writing out those unspoken, subconscious rules then the LLM has nothing to read about them in its training data.

Re: AI outperforms law professors in Stanford Law study

#72

Post by Esophagus4 »

ares623 wrote: post #44
Esophagus4 wrote: post #23

Yeah this could be interesting. A lot of the spotlight has been on “law firm stuff” like demand letters and writing contracts… But imagine if a dev team didn’t have to go engineer -> product manager -> legal team to get a question answered on local data retention requirements. You could ship that much faster.

Would you take responsibility for missing details about local data retention requirements?

Yes.

If the only purpose of asking a lawyer is transferring risk (aka cover your ass) while getting the same advice as an LLM, that’s slowing down delivery for purely bureaucratic reasons.

I’ve seen that mentality at big companies where everyone is scared to stick their neck out and be accountable for a decision. And nothing gets done. Drives me crazy.

But the people who move up are the people who take ownership and get shit done (and are right a lot).

(BTW, I have been at companies that were sued by regulators. They never really punish the individual(s) who were in the room when the decision is made. So your worry is kind of misplaced.)

Re: AI outperforms law professors in Stanford Law study

#73

Post by runarberg »

wilg wrote: post #52
aetq51 wrote: post #26

[flagged]

You're suspicious that the person doing academic research on how AI applies to law has a job related to research on law and AI?

You are not? It is at least worth investigating how much this professor benefits from AI companies. In fact this is HN. Let me come back to you in about 10 minutes.

EDIT: 10 min later. I give up. I tried to find who is funding HAI, and came empty handed, usually you can see that in their yearly reports, but no such luck for me. I know Google and Bill Gates are big donors, so take that as you will.

Re: AI outperforms law professors in Stanford Law study

#74

Post by applfanboysbgon »

jedberg wrote: post #65
zuzululu wrote: post #47

Earlier quoted context omitted.

honestly if you just avoid EU and China you can get away with anything

California too.

And with those three places listed you've ruled out literally 40% of the world economy. Great, you can ship your product in bumfuck Nebraska.

Re: AI outperforms law professors in Stanford Law study

#75

Post by Esophagus4 »

applfanboysbgon wrote: post #35
Esophagus4 wrote: post #27

Earlier quoted context omitted.

But isn’t this study saying that the legal advice could actually be better with AI? A bit of extrapolation from the study, but not a crazy stretch.

Maybe, although I would be extremely hesitant to extrapolate from this one study and trust my legal life to an LLM. One thing that's worth noting, though, is that regardless of the quality of objective legal advice in the abstract, for a lot of smaller scale stuff the human connection actually is literally what is important. There are ambiguities in the law, which are not resolved deterministically but rather at the…

Fair.

But I could also see a world where that, too, is fed to models for hyper-local results.

Could be a way off, but I could see it.

Re: AI outperforms law professors in Stanford Law study

#77

Post by xiaoyu2006 »

ChrisMarshallNY wrote: post #63
thewebguyd wrote: post #15

Earlier quoted context omitted.

I think this is probably true for most skilled professions. AI is best used in the hands of folks already knowledgeable in the skills/professions they are using it for. I liken it to me googling things as a sysadmin vs. Jane from accounting doing it. The non-tech end user is far more likely to make the problem worse, or install something sketchy from the ad riddled results than I am, or one of my help desk employees…

> I wouldn't really want to rely on my lawyer to use AI to write code for me. Yet that is exactly what a lot of C-Suiters (many of whom are lawyers), are doing.

Vice versa there is also a lot of irresponsible programmers doing stupid things with ai. Irresponsible people stay irresponsible, AI just make them more productive at being irresponsible.

Re: AI outperforms law professors in Stanford Law study

#78

Post by OhioMan2943 »

More great news from the prestigious university where 40% of students claim they are disabled

https://fortune.com/article/rise-in-elite-students-seeking-a...

and where they wanted to ban words such as "chief", "stupid", "karen" and "American"

https://reason.com/2022/12/21/stanford-elimination-harmful-l...

Re: AI outperforms law professors in Stanford Law study

#79

Post by _heimdall »

causal wrote: post #7

As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.

I wouldn't consider drafting a will to be harmless. If its done poorly the next of kin could have to deal with a huge headache and potentially months or years of probate proceedings.

Re: AI outperforms law professors in Stanford Law study

#80

Post by rayiner »

dylan604 wrote: post #68
rayiner wrote: post #22

Earlier quoted context omitted.

I would think that LLMs would be better at avoiding foot-guns. That’s a situation where you have a list of well known rules and potential pit falls, and the work of the lawyer is to apply those to a fact pattern. That’s something that has been hard to automate programmatically, because the fact patterns are similar but different. LLMs, however, seem to excel at applying general principles to differing fact patterns.

But can an LLM come up with questions like what the definition of is is? Seems to me there's a lot of "depends on how you read it" type of stuff that lawyers excel at finding novel interpretations. So what coders thinking of as rules are much less straight forward to understand when it comes to laws

I think that’s a different task than the one OP is referring to. To your example, I’m not familiar with the capability of LLMs in that regard. I have struggled with using the AI features of westlaw when it comes to that sort of argument. (Basically, making an argument that strays from typical route, because that’s the position you happen to find yourself representing.)
Post reply on HN