Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI outperforms law professors in Stanford Law study

law.stanford.edu

Post reply on HN
Print view View original

281–290 of 384 posts

Re: AI outperforms law professors in Stanford Law study

#281

Post by alansaber »

aristofun wrote: post #235

In general it is not surprising. Even if this particular study is bad. There are certain areas of law work that are about analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more. That is literally the bread of LLMs. Those types of lawyers should be the first in line for unemployment, not programmers, not even close.

"That is literally the bread of LLMs." correct. However, programming has a large number of advantages RE LLM use compare to law:

You can execute the logic, and set up loops from the output. You can set up more useful RL. It's easier to generate synthetic training data. It naturally supports tool use and agent parallelism. It's easier to integrate with APIs (with what few APIs the court systems provide). Programming explicitly encodes abstractions at the function, module levels etc that are easier to KG/reason/build upon than text chunks.

Re: AI outperforms law professors in Stanford Law study

#282

Post by wouldbecouldbe »

spider-mario wrote: post #246
wouldbecouldbe wrote: post #195

Earlier quoted context omitted.

The issue is, it almost always outperforms knowledge workers. IF the right questions are asked, and IF steered into and corrected at a few crucial points. IF not it goes off in the wrong direction really quick and that's a problem that's still mostly unsolved in the last 2 years. And that can be catastrophic in high risk environments, like legal, medical or high risk software products where being wrong in the wrong p…

> And that can be catastrophic in high risk environments, like legal, medical or high risk software products where being wrong in the wrong place can mean bankruptcy or even cost a life. Which also happens with humans – does it do so at a lower rate? On its own, it kind of sounds like similar anti-self-driving-car arguments.

yeah thats why I mentioned it works well IF guided by the correct expert.

I agree that you can create a set of domain specific rules, reinforcement layer validation tools, like self driving, that vastly improves the accuracy of au & llm's. Making humans less and less needed. But where LLM's comes from the magic of generic knowledge, this will be the opposite, narrowing it down.

Re: AI outperforms law professors in Stanford Law study

#283

Post by snk »

windexh8er wrote: post #249
byzantinegene wrote: post #194

Earlier quoted context omitted.

it's also worth keeping in mind that alot of the 'improvements' are actually advancements in harnesses and tools.

This is the hot button right here. Most of the advancements have also come at the cost of excess: exponential token use at the expense of marginal gains. Context is still a large limiting factor, and we have band aids around that area already. And the further along we go the further distributed LLMs get in terms of additional pieces. As for the original article and sentiment I'm sure AI will be a boon for law. It's g…

I'll note that having countless superintelligences handy will accelerate problem-solving e.g. dystopia.

Re: AI outperforms law professors in Stanford Law study

#284

Post by songting591 »

The interesting shift isn't whether AI beats law professors on tests — it's what happens to the value chain after that threshold is crossed.

When AI clears the knowledge bar in a domain, the remaining moat becomes trust, accountability, and local regulatory context. That's actually good news for niche SaaS builders targeting specific jurisdictions: the generic AI layer commoditizes, but the "AI + local compliance + human accountability" bundle still has real pricing power.

Curious whether anyone has seen this play out already in contract review or compliance tooling outside the US.

Re: AI outperforms law professors in Stanford Law study

#285

Post by [deleted] »

AlecSchueler wrote: post #203
gguncth wrote: post #186

Earlier quoted context omitted.

Sure, but in two years AI has gone from “impressive tool, but not a replacement for knowledge workers” to “the study where it beats our highest caliber of knowledge workers may have some methodological deficits.” In another two years it’s going to be curtains.

> the study where it beats our highest caliber of knowledge workers may have some methodological deficits The point is that if the study can't validate the claims being made then we can't actually extrapolate from that claim. What you're predicting may or may come true, but the study (which is the topic at hand) isn't useful for supporting the assertion.

[deleted]

Re: AI outperforms law professors in Stanford Law study

#286

Post by bobro »

gguncth wrote: post #186
godelski wrote: post #101

I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's als…

Sure, but in two years AI has gone from “impressive tool, but not a replacement for knowledge workers” to “the study where it beats our highest caliber of knowledge workers may have some methodological deficits.” In another two years it’s going to be curtains.

>the study where it beats our highest caliber of knowledge workers may have some methodological deficits.

That isn’t even remotely what this study is looking at.

Re: AI outperforms law professors in Stanford Law study

#287

Post by threethirtytwo »

acdha wrote: post #62
bko wrote: post #30

Marc Andreessen argued that we've already reached AGI. He says that the top AI models give better answers than 99% of people he has access to, and he has access to some of the best people in their field. I'm getting more convinced. I mean, sure it makes dumb mistakes sometimes but its a particular set of self serving mistakes, commenting out tests in order to pass. We obv don't want this behavior but I wouldn't say i…

> Marc Andreessen argued that we've already reached AGI. He says that the top AI models give better answers than 99% of people he has access to, and he has access to some of the best people in their field. He stands to make billions if enough people believe him — unless you also do, consider that you’re the mark. For example, if that was true, it would have to mean that AI companies either aren’t letting customers us…

And you stand to lose your job and your identity as a programmer.

He makes billions but he already is a billionaire. Gaining billions more doesn't mean shit. The guy really has nothing to lose and the utility of what he gains contribute little to his life style.

I will tell you this. HN has been comically wrong about everything related to AI. They said driverless cars have no chance of becoming useable. Now Tesla FSD is almost there and I sleep in waymo cars. HN said AI will never code, now everyone uses it to code.

It's fucking stupid. This is one of the smartest forums on the internet but HN becomes next to stupid when predicting AI. Why? Because humans can't face the truth. When the victim of attack is yourself, it doesn't matter how smart you are... you have to scaffold a rationalization to spare yourself as the victim. You have to lie to yourself and tell yourself that you matter.

The truth of it is, while LLMs are not the end game, AI in general is on a trajectory to take over. It shows us how meaningless our skills are... not only as programmers but as artists. That beautiful song you felt had greater meaning? It's all reproducible via an algorithm because it never really had a greater meaning. It was just a pattern.

Re: AI outperforms law professors in Stanford Law study

#288

Post by threethirtytwo »

moregrist wrote: post #59
bko wrote: post #30

Marc Andreessen argued that we've already reached AGI. He says that the top AI models give better answers than 99% of people he has access to, and he has access to some of the best people in their field. I'm getting more convinced. I mean, sure it makes dumb mistakes sometimes but its a particular set of self serving mistakes, commenting out tests in order to pass. We obv don't want this behavior but I wouldn't say i…

Marc Andreessen has a strong financial incentive to feel this way and to convince others to feel this way. I also think it’s easy to think that AI gives good answers if you don’t know the field well. In fields where I know the material, the answers are pretty variable and can be quite bad.

HNers have strong incentive to feel the opposite. Humanity in general has strong incentive to feel the opposite.

AI is not only replacing programmers, but art and the meaning of being human itself. It's showing us how trivial all of human creation is as it's just patterns from an algorithm.

Re: AI outperforms law professors in Stanford Law study

#289

Post by snk »

mdlman wrote: post #126
U4E4 wrote: post #116

Earlier quoted context omitted.

This is true in the naive case. There are however LLM context building techniques that anchor completions in data structures that persist the structure of claims that support the conclusion contained in a completion. Lots of different patterns exist —organizing logic in language is a rich domain— but the one I’ve liked the most is something called a Claim Dependency Graph that models the relationships between atomic…

I’d love to read more about these type of patterns. Do you have any recommendations?

https://arxiv.org/pdf/2404.14372
Post reply on HN