Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI outperforms law professors in Stanford Law study

law.stanford.edu

Post reply on HN
Print view View original

241–250 of 384 posts

Re: AI outperforms law professors in Stanford Law study

#241

Post by hparadiz »

15155 wrote: post #220
causal wrote: post #7

As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.

> drafting a will Tell me you've never been the executor of an estate in the United States without telling me.

I think going through this process has made me uniquely qualified to write one.

Re: AI outperforms law professors in Stanford Law study

#242

Post by nopurpose »

Forgeties79 wrote: post #213
nopurpose wrote: post #193

Earlier quoted context omitted.

On the other hand we keep seeing only marginal generational imorovements in CPU space, yet performance gains over last 10 years in CPUs are very material. Every new model might not be a leap like it used to be, but give it enough time and improvements add up.

Nobody is disputing that. I specifically said that I can see the improvements from the last six months. What I’m saying is we can’t assume that every two years it will improve at the same rate. The further we get into this, the more AI feels like 3-D printing. Significantly bigger and will be more widely used for sure. But nowhere near the “new industrial revolution” that all these companies are making it out to be

Do you agree that economic and behaviour shift will be comparable to mobile and we are at the times of Nokia 3310. Does it count as industrial revolution?

Re: AI outperforms law professors in Stanford Law study

#243

Post by KaiserPro »

pezgrande wrote: post #227
wouldbecouldbe wrote: post #195

Earlier quoted context omitted.

The issue is, it almost always outperforms knowledge workers. IF the right questions are asked, and IF steered into and corrected at a few crucial points. IF not it goes off in the wrong direction really quick and that's a problem that's still mostly unsolved in the last 2 years. And that can be catastrophic in high risk environments, like legal, medical or high risk software products where being wrong in the wrong p…

I kinda disagree. High risk environments just means that they will have to have a human-in-the-loop for a longer time which drastically reduce the skill required for such human (which is still requires high skill just not stupidly high).

> drastically reduce the skill required for such human

I mean thats what is wanted by some companies.

The problem, especially for things like legal is that it requires someone more skilled to read through and understand that the argument is bollocks, or the law/precedent they are banking on is in fact the right one.

We have a tool that auto-writes letters to our management companies when they break SLAs. We have a slider that goes from polite to we are going to extract your first born.

Thats simple ish to do for LLMs, and low risk.

Drafting contracts is also something we could probably do, as its mostly boilerplate. However the consequence for mis-drafting a contract is multi-million dollars.

Re: AI outperforms law professors in Stanford Law study

#244

Post by hparadiz »

gaiagraphia wrote: post #56

Incredible that the common people will be able to wrestle the right to rule of law away from the bloated legal caste, who have built themselves quite the moat. The inaccessibility of justice is a huge driver of inequality. Any tools which bridge this gap will help make a more just society.

The profession is walking into a court room 90 minutes late because you know the judge's work pattern then going "hey Mike, how are the kids" after 22 years in the same jurisdiction. Then they old boys haggle based on how much the lawyer is charging. You are basically paying for access to the social club. Better outcomes when part of the in-group of course.

Re: AI outperforms law professors in Stanford Law study

#245

Post by goodcanadian »

mchl-mumo wrote: post #150
quantisan wrote: post #96

I'm surprised Stanford Law would go along with this over-reaching press release title. How about "For common first-year contracts-law questions, law professors preferred AI-generated answers to professor-generated answers"

The revised title is spot on. It's odd to me how academics are trying to sound like top research labs' CEOs trying to pump valuations by overreaching claims.

It is rarely the academics writing the press release. It is even rarer that the author of the press release chooses the title.

Re: AI outperforms law professors in Stanford Law study

#246

Post by spider-mario »

wouldbecouldbe wrote: post #195
gguncth wrote: post #186

Earlier quoted context omitted.

Sure, but in two years AI has gone from “impressive tool, but not a replacement for knowledge workers” to “the study where it beats our highest caliber of knowledge workers may have some methodological deficits.” In another two years it’s going to be curtains.

The issue is, it almost always outperforms knowledge workers. IF the right questions are asked, and IF steered into and corrected at a few crucial points. IF not it goes off in the wrong direction really quick and that's a problem that's still mostly unsolved in the last 2 years. And that can be catastrophic in high risk environments, like legal, medical or high risk software products where being wrong in the wrong p…

> And that can be catastrophic in high risk environments, like legal, medical or high risk software products where being wrong in the wrong place can mean bankruptcy or even cost a life.

Which also happens with humans – does it do so at a lower rate? On its own, it kind of sounds like similar anti-self-driving-car arguments.

Re: AI outperforms law professors in Stanford Law study

#247

Post by Paradigma11 »

xattt wrote: post #94
applicative wrote: post #93

What the LLM cannot do is explain why it said what it said, when cross-examined. It simply hallucinates the best account of why someone would have said such a thing as it said, same as it can give a probable account of why someone else said something different. The question 'But why did you say this not that ...?' does not lead it to make explicit its grounds for what it said, but just to make a new more complicated…

A human has a motive that exists that frames the thought being expressed. An LLM is going to be creating a “de novo” thought in response to a line of questioning.

Psychology has shown that a lot of those motives are just post hoc narratives, similar to LLM.

Re: AI outperforms law professors in Stanford Law study

#248

Post by Forgeties79 »

nopurpose wrote: post #242
Forgeties79 wrote: post #213

Earlier quoted context omitted.

Nobody is disputing that. I specifically said that I can see the improvements from the last six months. What I’m saying is we can’t assume that every two years it will improve at the same rate. The further we get into this, the more AI feels like 3-D printing. Significantly bigger and will be more widely used for sure. But nowhere near the “new industrial revolution” that all these companies are making it out to be

Do you agree that economic and behaviour shift will be comparable to mobile and we are at the times of Nokia 3310. Does it count as industrial revolution?

I think that’s kind of a strange question/parallel that doesn’t have a concrete answer, partially because even the people making these tools don’t really know where it’s going to land or what the ultimate utility is. Hence why they’re begging all of us to figure out the billion dollar applications for them.

Ultimately they are clearly here to stay but I think they are going to be incredibly important in some industries and minimally present in others (a glorified chatbot/summarizing tool for instance). Whatever form it takes it’s definitely not going to be a model where individuals have subscriptions they pay for monthly.

Re: AI outperforms law professors in Stanford Law study

#249

Post by windexh8er »

byzantinegene wrote: post #194
Forgeties79 wrote: post #190

Earlier quoted context omitted.

Assuming it keeps improving at the same rate, which I think we are already seeing not play out. If you compare the first six months when GPT truly hit the mainstream to the previous six months, the improvements are not nearly as evident. That isn’t to say they aren’t noticeable, I could definitely tell it’s improving, but not nearly at the pace it once was. There’s also the fact that they can’t possibly keep improvin…

it's also worth keeping in mind that alot of the 'improvements' are actually advancements in harnesses and tools.

This is the hot button right here. Most of the advancements have also come at the cost of excess: exponential token use at the expense of marginal gains.

Context is still a large limiting factor, and we have band aids around that area already. And the further along we go the further distributed LLMs get in terms of additional pieces.

As for the original article and sentiment I'm sure AI will be a boon for law. It's going to be much easier for the general consumer / person / small business to represent themselves which feels like a win. The downside is I feel like we're tracking towards a digital hell of "virtual lawyers" that will be at the whim of any org. Consumer laws really need to change now to help avoid this dystopian path we're on.

Re: AI outperforms law professors in Stanford Law study

#250

Post by IFC_LLC »

This is exactly what LLM designed to do. Double up a lot of data and find connections and patterns in it.

So no wonder on this point.

One thing I want to mention: Law != Justice.

So while LLMs are awesome at the law study they will suck at justice. Just because one has to solve very emotional problems with it at times. And LLMs are not that good at finding the correct emotion.

Post reply on HN