Live data from Hacker News

AI outperforms law professors in Stanford Law study

law.stanford.edu

21–30 of 384 posts

Re: AI outperforms law professors in Stanford Law study

#21
post #13
post #10

Earlier quoted context omitted.

I do wish they'd used some more objective criteria. Simply being preferable one of the things LLMs have trained for since the beginning, hence its sycophantic nature.

What criteria would you use for judging legal arguments?

maybe seeing if the case law it cited was real or imagined? Just one idea, IANAL

Re: AI outperforms law professors in Stanford Law study

#22
post #7

As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.

I would think that LLMs would be better at avoiding foot-guns. That’s a situation where you have a list of well known rules and potential pit falls, and the work of the lawyer is to apply those to a fact pattern. That’s something that has been hard to automate programmatically, because the fact patterns are similar but different. LLMs, however, seem to excel at applying general principles to differing fact patterns.

Re: AI outperforms law professors in Stanford Law study

#23
Yeah this could be interesting. A lot of the spotlight has been on “law firm stuff” like demand letters and writing contracts…

But imagine if a dev team didn’t have to go engineer -> product manager -> legal team to get a question answered on local data retention requirements. You could ship that much faster.

Re: AI outperforms law professors in Stanford Law study

#27

Earlier quoted context omitted.

If you want human connection the legal system is not where you are going to find it, period. I don't think there will be any such market for "non ai" law. If I'm involved with the legal system I just want out as quick as possible as cheap as possible.

Bad legal advice will keep you dealing with the legal system for much longer and at much greater cost. Something being cheap and quick upfront doesn't mean it will be cheap and quick by the end of the process.

But isn’t this study saying that the legal advice could actually be better with AI?

A bit of extrapolation from the study, but not a crazy stretch.

Re: AI outperforms law professors in Stanford Law study

#28
post #3

I think there will be a market for firms that aggressively market themselves as non-AI, and then as more people turn towards that human connection we'll go full circle

Nobody wants to pay their lawyers more than they have to. There will be a huge market for firms that can use AI to avoid charging clients for $1,000/hour junior associates.

Re: AI outperforms law professors in Stanford Law study

#29
post #7

As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.

there’s really no limit to how many times and ways you can review something with AI, except dollars.

Re: AI outperforms law professors in Stanford Law study

#30
Marc Andreessen argued that we've already reached AGI. He says that the top AI models give better answers than 99% of people he has access to, and he has access to some of the best people in their field.

I'm getting more convinced. I mean, sure it makes dumb mistakes sometimes but its a particular set of self serving mistakes, commenting out tests in order to pass. We obv don't want this behavior but I wouldn't say it's dumb.

It'll be like the Turing test, which we just blew past years ago and no one cared. After all the hand-wringing about sentience and rights of the AI if it passes the Turing test, and now we just have AI bots running 24/7 writing slop.

How does everyone else feel?

Post reply on HN