Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI outperforms law professors in Stanford Law study

law.stanford.edu

Post reply on HN
Print view View original

251–260 of 384 posts

Re: AI outperforms law professors in Stanford Law study

#251

Post by streetfighter64 »

aristofun wrote: post #235

In general it is not surprising. Even if this particular study is bad. There are certain areas of law work that are about analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more. That is literally the bread of LLMs. Those types of lawyers should be the first in line for unemployment, not programmers, not even close.

> analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more

The same could be said about programming. Or if you want to be even more reductive, looking at a screen and pressing buttons to make the correct lights light up https://xkcd.com/722/

Re: AI outperforms law professors in Stanford Law study

#252

Post by coldtea »

IFC_LLC wrote: post #250

This is exactly what LLM designed to do. Double up a lot of data and find connections and patterns in it. So no wonder on this point. One thing I want to mention: Law != Justice. So while LLMs are awesome at the law study they will suck at justice. Just because one has to solve very emotional problems with it at times. And LLMs are not that good at finding the correct emotion.

Also because their reasoning is just a statistical model of whatever they've been fed. No experience of pain, humility, human connection, etc in this.

Re: AI outperforms law professors in Stanford Law study

#253

Post by coldtea »

bko wrote: post #30

Marc Andreessen argued that we've already reached AGI. He says that the top AI models give better answers than 99% of people he has access to, and he has access to some of the best people in their field. I'm getting more convinced. I mean, sure it makes dumb mistakes sometimes but its a particular set of self serving mistakes, commenting out tests in order to pass. We obv don't want this behavior but I wouldn't say i…

>Marc Andreessen argued that we've already reached AGI. He says that the top AI models give better answers than 99% of people he has access to, and he has access to some of the best people in their field.

He has access to employees and yes-men. What he actually needs to hear, nobody will tell him, AI even less so. Every shit idea he has, would be "what a bright idea"-ed by both everyone around him and AI.

And of course there's the little matter that he makes money and increases his power by selling AI. What seller doesn't promote their stuff as the greatest ever?

Re: AI outperforms law professors in Stanford Law study

#254

Post by conartist6 »

aristofun wrote: post #235

In general it is not surprising. Even if this particular study is bad. There are certain areas of law work that are about analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more. That is literally the bread of LLMs. Those types of lawyers should be the first in line for unemployment, not programmers, not even close.

I see the same problem with AI in both programming and law though.

AI is like a scab on a wound: it's a temporary filler, it rushes in to fill a void, but it's not going to be the final solution.

Models showed us that there was huuuge unmet demand for literacy, both in software and in law. But now we have a choice to either address the systemic causes of the unmet demand, or just try to paper over them with layers and layers of AI scab.

Re: AI outperforms law professors in Stanford Law study

#255

Post by tiahura »

mxkopy wrote: post #177
em500 wrote: post #162

Earlier quoted context omitted.

> The biggest trap is the hallucinated citation. It will easily insert an absolutely authentic sounding quotation from another case that perfectly proves the point you are trying to make, then it'll make up an authentic name for it, e.g. United States v. Shenzhou Electronics Inc or whatever. Naive question from an outsider: aren't there searchable databases of cases (with complete text) so that citations could be che…

It’s a band aid solution because the model can get stuck in a refutation loop, where it argues a point by pulling up a contradicting source ad infinitum. The holy grail, which has not been yet reached, is figuring out how to dynamically align the model to be consistent with all the sources in the first place (and this is a problem of provenance rather than model design)

I’ve been doing ai legal research via caselaw api with Claude code for at least a year and I’ve never seen that happen.

Re: AI outperforms law professors in Stanford Law study

#256

Post by epolanski »

IshKebab wrote: post #183
esquivalience wrote: post #160

Earlier quoted context omitted.

I think your 3k figure comes from here - It is explained: > As judges, the professors then completed 2,918 blinded, forced-choice comparisons (median per judge: 200), each time indicating which of the two anonymized responses, from the instructor or the LLM, they would rather give to a student

So did were the answers fact checked? If not that seems like a pretty obvious flaw!

The study deliberately analyzes questions that don't have clear black or white answers, what matters is the reasoning.

Re: AI outperforms law professors in Stanford Law study

#257

Post by tiahura »

timpera wrote: post #175
BartjeD wrote: post #164

Earlier quoted context omitted.

A legal professional can be personally liable for not finding the most recent case-law. The knowledge cut off gap means the models sometimes don't know about the most recent case-law, in a given situation. I've seent his happen multiple times now. Accountants and legal professionals advising clients based on outdated information assembled through chat-gtp, claude and copilot. Professionals drafting letters and missin…

You definitely want your AI to search legal databases, and not draw from "memory". This is where AI offerings from Thomson or Lexis could shine, especially in jurisdictions where case law is not freely available online.

Or you can just have Claude code search westlaw / vlex/ courtlistener

Re: AI outperforms law professors in Stanford Law study

#258

Post by epolanski »

aristofun wrote: post #235

In general it is not surprising. Even if this particular study is bad. There are certain areas of law work that are about analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more. That is literally the bread of LLMs. Those types of lawyers should be the first in line for unemployment, not programmers, not even close.

The more I see the evolution, the more it looks to me that any knowledge workers is going to be impacted.

Re: AI outperforms law professors in Stanford Law study

#259

Post by tiahura »

ChrisMarshallNY wrote: post #63
thewebguyd wrote: post #15

Earlier quoted context omitted.

I think this is probably true for most skilled professions. AI is best used in the hands of folks already knowledgeable in the skills/professions they are using it for. I liken it to me googling things as a sysadmin vs. Jane from accounting doing it. The non-tech end user is far more likely to make the problem worse, or install something sketchy from the ad riddled results than I am, or one of my help desk employees…

> I wouldn't really want to rely on my lawyer to use AI to write code for me. Yet that is exactly what a lot of C-Suiters (many of whom are lawyers), are doing.

Corporations are DEMANDING legal ai because it is so much more efficient.

Re: AI outperforms law professors in Stanford Law study

#260

Post by [deleted] »

applicative wrote: post #93

What the LLM cannot do is explain why it said what it said, when cross-examined. It simply hallucinates the best account of why someone would have said such a thing as it said, same as it can give a probable account of why someone else said something different. The question 'But why did you say this not that ...?' does not lead it to make explicit its grounds for what it said, but just to make a new more complicated…

[deleted]
Post reply on HN