Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI outperforms law professors in Stanford Law study

law.stanford.edu

Post reply on HN
Print view View original

311–320 of 384 posts

Re: AI outperforms law professors in Stanford Law study

#311

Post by ambicapter »

pezgrande wrote: post #227
wouldbecouldbe wrote: post #195

Earlier quoted context omitted.

The issue is, it almost always outperforms knowledge workers. IF the right questions are asked, and IF steered into and corrected at a few crucial points. IF not it goes off in the wrong direction really quick and that's a problem that's still mostly unsolved in the last 2 years. And that can be catastrophic in high risk environments, like legal, medical or high risk software products where being wrong in the wrong p…

I kinda disagree. High risk environments just means that they will have to have a human-in-the-loop for a longer time which drastically reduce the skill required for such human (which is still requires high skill just not stupidly high).

Doesn't it increase the skill required? You need to be able to jump in at the perfect time, while waiting patiently for 99% of the time. It's like self-driving that requires you to "jump in" at the worst possible time (0.5 seconds from a crash), and stay put the rest of the time--but don't get bored or inattentive. The only way to do that would be to be so naturally good at the danger point that you can do it basically reflexively.

Re: AI outperforms law professors in Stanford Law study

#312

Post by [deleted] »

ToValueFunfetti wrote: post #297
ryan_n wrote: post #234

Earlier quoted context omitted.

> Just a humble opinion that I would love to see be wrong Out of curiosity, why would you love to be wrong about that? What possible outcome could you see being a net positive for society if the vast majority of knowledge workers (and ultimately, as robotics progress, most workers in general) are replaced by AI?

I believe it was Blink-182 who said, "Work sucks". You have to pay people to do that stuff; they don't want to be there. And then you get into second order effects- costs plummet for anything labor intensive, including medical care, prepared food, cleaning, and private tutors. Then onto tertiary effects- if you can spin up a million genius researchers to attack a problem, you start seeing massive progress in every im…

[deleted]

Re: AI outperforms law professors in Stanford Law study

#313

Post by giancarlostoro »

godelski wrote: post #101

I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's als…

I never get the same answer from any two lawyers. I hate law as a result. With developers you might get disagreements based on experience, but there's usually a strong consensus on specific things, with lawyers and courts its all over the flipping place. I wouldn't be surprised if LLMs can "pass" on paper (ie college exams) but in practice, they might 'struggle' in different courts.

...On the other hand, if an LLM has access to every transcript of every case a Judge has overseen, they might have an unfair advantage in any case... Hmmm...

This all assuming the AI lawyer doesn't hallucinate and start referencing cases that don't exist.

Re: AI outperforms law professors in Stanford Law study

#317

Post by acdha »

threethirtytwo wrote: post #287
acdha wrote: post #62

Earlier quoted context omitted.

> Marc Andreessen argued that we've already reached AGI. He says that the top AI models give better answers than 99% of people he has access to, and he has access to some of the best people in their field. He stands to make billions if enough people believe him — unless you also do, consider that you’re the mark. For example, if that was true, it would have to mean that AI companies either aren’t letting customers us…

And you stand to lose your job and your identity as a programmer. He makes billions but he already is a billionaire. Gaining billions more doesn't mean shit. The guy really has nothing to lose and the utility of what he gains contribute little to his life style. I will tell you this. HN has been comically wrong about everything related to AI. They said driverless cars have no chance of becoming useable. Now Tesla FSD…

> Gaining billions more doesn't mean shit. The guy really has nothing to lose and the utility of what he gains contribute little to his life style.

You don’t become a billionaire because you aren’t committed to making a number go up far after you no longer have any significant unmet needs. He’s spending his life focused on business deals because that’s what he cares most about — if his true love was science, philanthropy, etc. he’d have been able to do that full time a couple decades ago.

Re: AI outperforms law professors in Stanford Law study

#318

Post by saidnooneever »

godelski wrote: post #101

I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's als…

more and more i see papers. interview 8 ppl, draw conclusions based on their expert opinions. AI and Cybersecurity are full of this.

Even saw some where they just slapped interviews + protocol into chatgpt as 'methodology' to extract the results -_-. Peer reviewed and published.

Re: AI outperforms law professors in Stanford Law study

#319

Post by godelski »

Paracompact wrote: post #146
godelski wrote: post #101

I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's als…

Independent of whether it has any meaning (because the entire paper might be a bit iffy), I find it curious that Instructors 3 and 8 have the lowest harmfulness rates, quite a bit lower than even the LLMs, but not the highest preference rates. Harmfulness anticorrelates with preference, but not perfectly. Some amount of charisma appears to be a factor even in selections by professionals?

Yeah it's difficult to interpret.

One possible interpretation, the statements were very bland. These would be very low harm but also not very informative

Re: AI outperforms law professors in Stanford Law study

#320

Post by iterance »

aristofun wrote: post #235

In general it is not surprising. Even if this particular study is bad. There are certain areas of law work that are about analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more. That is literally the bread of LLMs. Those types of lawyers should be the first in line for unemployment, not programmers, not even close.

Just because it is theoretically the bread and butter of LLMs does not mean LLMs are capable of doing the job. It still needs to be proven, setting prior beliefs aside. Law is a life-critical system and deserves our highest level of scrutiny.
Post reply on HN