Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI outperforms law professors in Stanford Law study

law.stanford.edu

Post reply on HN
Print view View original

291–300 of 384 posts

Re: AI outperforms law professors in Stanford Law study

#291

Post by kenjackson »

Forgeties79 wrote: post #190
gguncth wrote: post #186

Earlier quoted context omitted.

Sure, but in two years AI has gone from “impressive tool, but not a replacement for knowledge workers” to “the study where it beats our highest caliber of knowledge workers may have some methodological deficits.” In another two years it’s going to be curtains.

Assuming it keeps improving at the same rate, which I think we are already seeing not play out. If you compare the first six months when GPT truly hit the mainstream to the previous six months, the improvements are not nearly as evident. That isn’t to say they aren’t noticeable, I could definitely tell it’s improving, but not nearly at the pace it once was. There’s also the fact that they can’t possibly keep improvin…

The issue is that before GPT models basically were useless for any conversation. We are literally in science fiction realm. From a text conversation perspective the gap between where we are at and what’s left to get to is relatively small.

In my opinion, the main thing we need to do is have training happen continuously. And probably more real world data (from sensors).

Re: AI outperforms law professors in Stanford Law study

#292

Post by [deleted] »

nickburns wrote: post #280
aristofun wrote: post #235

In general it is not surprising. Even if this particular study is bad. There are certain areas of law work that are about analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more. That is literally the bread of LLMs. Those types of lawyers should be the first in line for unemployment, not programmers, not even close.

'Bread *and butter'. The English expression requires the second part—but otherwise fits perfectly in your well-stated point, with which I wholeheartedly agree. Source: AAL.

[deleted]

Re: AI outperforms law professors in Stanford Law study

#293

Post by greggoB »

jstummbillig wrote: post #214
greggoB wrote: post #181

Earlier quoted context omitted.

> I wonder how much we are learning with these studies about what is going on. So your alternative is to not have any studies and everyone can just stump up anecdata as "evidence" for the capabilities of these models?

Doing things that are well meaning, but ineffective is not great policy. The simplest alternative to doing things that don't work is always not doing them. Better ideas are of course welcome, but not required.

I don't think that's how science/academia works. There is no such thing as a perfect study, there are always non-idealities and noise in the data. Good studies make well-justified efforts to account for these, OP is saying they don't believe this is the case here.

Regardless, your assertion that "oh well, the models will be totally different in a few months anyway, therefore any study done today is pointless" seems more than a stretch. How do you know they will be so different? How can you verify that today's studies are completely irrelevant?

Re: AI outperforms law professors in Stanford Law study

#294

Post by jvvw »

pezgrande wrote: post #227
wouldbecouldbe wrote: post #195

Earlier quoted context omitted.

The issue is, it almost always outperforms knowledge workers. IF the right questions are asked, and IF steered into and corrected at a few crucial points. IF not it goes off in the wrong direction really quick and that's a problem that's still mostly unsolved in the last 2 years. And that can be catastrophic in high risk environments, like legal, medical or high risk software products where being wrong in the wrong p…

I kinda disagree. High risk environments just means that they will have to have a human-in-the-loop for a longer time which drastically reduce the skill required for such human (which is still requires high skill just not stupidly high).

The employers will think it requires less skill, whereas in fact it might actually require more skill to do a good job of being the human-in-the-loop.

For example, my sister is a translator and she says that checking AI translations is actually harder in many ways than doing a translation in the first place, but the agencies pay less for checking than actual translation.

Re: AI outperforms law professors in Stanford Law study

#295

Post by nopurpose »

Forgeties79 wrote: post #248
nopurpose wrote: post #242

Earlier quoted context omitted.

Do you agree that economic and behaviour shift will be comparable to mobile and we are at the times of Nokia 3310. Does it count as industrial revolution?

I think that’s kind of a strange question/parallel that doesn’t have a concrete answer, partially because even the people making these tools don’t really know where it’s going to land or what the ultimate utility is. Hence why they’re begging all of us to figure out the billion dollar applications for them. Ultimately they are clearly here to stay but I think they are going to be incredibly important in some industri…

> even the people making these tools don’t really know where it’s going to land

exactly my point to compare it with pre-iPhone mobile market: wide (and growing fast!) adoption, clear potential (WAP websites, J2ME games), many players in the game, some real market fit discovered already (Blackberry), influx of capitial and tinkerers alike, but still a lot of unknowns where it will ultimately land.

Even if no single improvement was revolutionary (even first iPhone was just a fancy phone without App Store), overall mobile made billion dollar industries possible, for better or worse, and changed the way we live. Counts as industrial revolution, comparable to the Internet itself in my eyes.

Re: AI outperforms law professors in Stanford Law study

#296

Post by the_gastropod »

kenjackson wrote: post #291
Forgeties79 wrote: post #190

Earlier quoted context omitted.

Assuming it keeps improving at the same rate, which I think we are already seeing not play out. If you compare the first six months when GPT truly hit the mainstream to the previous six months, the improvements are not nearly as evident. That isn’t to say they aren’t noticeable, I could definitely tell it’s improving, but not nearly at the pace it once was. There’s also the fact that they can’t possibly keep improvin…

The issue is that before GPT models basically were useless for any conversation. We are literally in science fiction realm. From a text conversation perspective the gap between where we are at and what’s left to get to is relatively small. In my opinion, the main thing we need to do is have training happen continuously. And probably more real world data (from sensors).

The ELIZA effect has been around since 1966. I think lots of folks feel “AI” has advanced much more quickly that it really has because of the nature of its many past boom / bust cycles.

Re: AI outperforms law professors in Stanford Law study

#297

Post by ToValueFunfetti »

ryan_n wrote: post #234
goolz wrote: post #198

Earlier quoted context omitted.

Ya, while the tools are really solid and have seen huge leaps these past two years, in no way will an LLM be able to do any of it unguided in two years. Just a humble opinion that I would love to see be wrong.

> Just a humble opinion that I would love to see be wrong Out of curiosity, why would you love to be wrong about that? What possible outcome could you see being a net positive for society if the vast majority of knowledge workers (and ultimately, as robotics progress, most workers in general) are replaced by AI?

I believe it was Blink-182 who said, "Work sucks". You have to pay people to do that stuff; they don't want to be there. And then you get into second order effects- costs plummet for anything labor intensive, including medical care, prepared food, cleaning, and private tutors. Then onto tertiary effects- if you can spin up a million genius researchers to attack a problem, you start seeing massive progress in every important area and it isn't tied to population growth.

I get that you might have a 'UBI/alternative general welfare is impossible' up your sleeve, but you've written this like it's somehow unfathomable that not forcing everybody to work just to survive would be a good thing. Of course it would be good! It's just a matter of dealing with the (huge) side effect of lost income.

Re: AI outperforms law professors in Stanford Law study

#298

Post by pezgrande »

moffkalast wrote: post #233
pezgrande wrote: post #227

Earlier quoted context omitted.

I kinda disagree. High risk environments just means that they will have to have a human-in-the-loop for a longer time which drastically reduce the skill required for such human (which is still requires high skill just not stupidly high).

If the human involved has no skill then they might as well not be there, since they're just a fall guy when things go wrong and won't do anything to prevent it from happening.

I said that still requires skill, just not as much.

Re: AI outperforms law professors in Stanford Law study

#299

Post by aristofun »

nickburns wrote: post #280
aristofun wrote: post #235

In general it is not surprising. Even if this particular study is bad. There are certain areas of law work that are about analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more. That is literally the bread of LLMs. Those types of lawyers should be the first in line for unemployment, not programmers, not even close.

'Bread *and butter'. The English expression requires the second part—but otherwise fits perfectly in your well-stated point, with which I wholeheartedly agree. Source: AAL.

Thank you! As a non native speaker I was not sure if “and butter” is a mandatory part but didn’t want (nor had time) to llm the comment for the sake of authenticity :) TIL

Re: AI outperforms law professors in Stanford Law study

#300

Post by aristofun »

streetfighter64 wrote: post #251
aristofun wrote: post #235

In general it is not surprising. Even if this particular study is bad. There are certain areas of law work that are about analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more. That is literally the bread of LLMs. Those types of lawyers should be the first in line for unemployment, not programmers, not even close.

> analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more The same could be said about programming. Or if you want to be even more reductive, looking at a screen and pressing buttons to make the correct lights light up https://xkcd.com/722/

Philosophically or metaphorically speaking - yes.

But in my comment it is literally what some subset of lawyers do.

Literally is much more tangible and risky in terms of real impact on employment etc.

Post reply on HN