Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI outperforms law professors in Stanford Law study

law.stanford.edu

Post reply on HN
Print view View original

271–280 of 384 posts

Re: AI outperforms law professors in Stanford Law study

#271

Post by ubercore »

stackghost wrote: post #67
thewebguyd wrote: post #15

Earlier quoted context omitted.

I think this is probably true for most skilled professions. AI is best used in the hands of folks already knowledgeable in the skills/professions they are using it for. I liken it to me googling things as a sysadmin vs. Jane from accounting doing it. The non-tech end user is far more likely to make the problem worse, or install something sketchy from the ad riddled results than I am, or one of my help desk employees…

It's like that in engineering, for sure. My background is in aerospace and there are lots of things that a reasonably technically-inclined random can probably do passably. It takes an engineer to know which tasks those are, though. I would imagine it's similar in law, in that it takes a lawyer or judge to know where the foot guns lie.

Agreed, and it's the same in software. Probably the biggest time-sink right now as a tech lead is people going from idea to fully-fleshed-out PR, and then having to go back to have a discussion of "was this the right thing to do". It causes frustration all around (being a "no" much more, and having someone tell you your finished work isn't valuable).

Re: AI outperforms law professors in Stanford Law study

#272

Post by gaiagraphia »

hparadiz wrote: post #244
gaiagraphia wrote: post #56

Incredible that the common people will be able to wrestle the right to rule of law away from the bloated legal caste, who have built themselves quite the moat. The inaccessibility of justice is a huge driver of inequality. Any tools which bridge this gap will help make a more just society.

The profession is walking into a court room 90 minutes late because you know the judge's work pattern then going "hey Mike, how are the kids" after 22 years in the same jurisdiction. Then they old boys haggle based on how much the lawyer is charging. You are basically paying for access to the social club. Better outcomes when part of the in-group of course.

Would like to plot attitudes to AI against parental incomes or inheritance. If your value derives from having contacts and access to gatekept materials, rather than pure technical expertise, you've got a lot to lose as the walls come crumbling down.

There was another thread about the impact of AI on maths, and one of the arguments was about peer review... Made me wonder whether the writer was more concerned about the established order and gates being upset, or whether there's actually a valid technical criticism.

Re: AI outperforms law professors in Stanford Law study

#273

Post by skinfaxi »

qingcharles wrote: post #152
causal wrote: post #7

As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.

I've used general purpose LLM AI (e.g. run-of-the-mill Claude, GPT etc) heavily to draft legal documents. The biggest trap is the hallucinated citation. It will easily insert an absolutely authentic sounding quotation from another case that perfectly proves the point you are trying to make, then it'll make up an authentic name for it, e.g. United States v. Shenzhou Electronics Inc or whatever. You can get really comf…

Chatgpt regularly hallucinates entire cases whole cloth or fabricates an entirely different fact pattern for a given case. Perplexity does much better at citing its sources and providing accurate quotes, at least in my experience.

Re: AI outperforms law professors in Stanford Law study

#274

Post by scotty79 »

pezgrande wrote: post #227
wouldbecouldbe wrote: post #195

Earlier quoted context omitted.

The issue is, it almost always outperforms knowledge workers. IF the right questions are asked, and IF steered into and corrected at a few crucial points. IF not it goes off in the wrong direction really quick and that's a problem that's still mostly unsolved in the last 2 years. And that can be catastrophic in high risk environments, like legal, medical or high risk software products where being wrong in the wrong p…

I kinda disagree. High risk environments just means that they will have to have a human-in-the-loop for a longer time which drastically reduce the skill required for such human (which is still requires high skill just not stupidly high).

The end game of this is just a human capable of taking the blame when AI makes an occasional mistake and being paid for that service and risk.

Re: AI outperforms law professors in Stanford Law study

#275

Post by scotty79 »

aristofun wrote: post #235

In general it is not surprising. Even if this particular study is bad. There are certain areas of law work that are about analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more. That is literally the bread of LLMs. Those types of lawyers should be the first in line for unemployment, not programmers, not even close.

LLMs answered student questions of the top of their heads, without any refresher look into the case law. And systems that were primied with the case law like NotebookLM underperformed when compared to baseline LLMs that you'd as anything about anything.

It's not about what LLMs can or are suited to do. This study shows strengths of what's already in them, innately.

Re: AI outperforms law professors in Stanford Law study

#276

Post by bluefirebrand »

koe123 wrote: post #238
ryan_n wrote: post #234

Earlier quoted context omitted.

> Just a humble opinion that I would love to see be wrong Out of curiosity, why would you love to be wrong about that? What possible outcome could you see being a net positive for society if the vast majority of knowledge workers (and ultimately, as robotics progress, most workers in general) are replaced by AI?

In a way, we are betraying something here. My reading is: solving the social problems of capitalism feels so impossible, that reducing the need for anyone to do work is a liability. In a way this sentiment should make extremists of us all?

Unfortunately the fact is that society has some massive imbalances around capitalism

It is not hard for me to imagine a world where if my bosses didn't need me, they would prefer me to be dead than to pay me some kind of permanent income to me. They would prefer to keep that power to themselves

These are already the sort of people who will happily lay you off into a recession, leave you without a way to pay your rent or for food if it improves their bottom line. They do not care if you starve. Or at least they care less than they do about their quarterly bonus

So no, I don't trust these fucks to continue playing nice if they view my value as going to zero

Re: AI outperforms law professors in Stanford Law study

#277

Post by nonethewiser »

wouldbecouldbe wrote: post #195
gguncth wrote: post #186

Earlier quoted context omitted.

Sure, but in two years AI has gone from “impressive tool, but not a replacement for knowledge workers” to “the study where it beats our highest caliber of knowledge workers may have some methodological deficits.” In another two years it’s going to be curtains.

The issue is, it almost always outperforms knowledge workers. IF the right questions are asked, and IF steered into and corrected at a few crucial points. IF not it goes off in the wrong direction really quick and that's a problem that's still mostly unsolved in the last 2 years. And that can be catastrophic in high risk environments, like legal, medical or high risk software products where being wrong in the wrong p…

Yeah but even what you describe makes it an extremely useful tool and productivity boost. Sure, we're not going to deploy a lawyer agent with full autonomy and no more oversight than a real lawyer. But isn't it wild that's now the frontier?

It's not like self driving cars where better than a human 80% of the time isn't good enough and they aren't really usable until its 95%, 99% etc.

Re: AI outperforms law professors in Stanford Law study

#278

Post by scotty79 »

finnborge wrote: post #120

I understand why the conversation on this article looks like it does, but the study is specifically focused on the potential for LLMs to operate as tutors for law students. I enjoy the extrapolation out to whether LLMs will replace lawyers, but did not find that to be discussed in the study itself. In the framing of using LLMs as legal tutors, with the implication of lowering the cost of legal training, this seems li…

> On the other hand, I do _not_ think that this study provides any indication that an LLM is prepared to actually provide direct legal counsel

I think it indicates that LLMs are smart enough to be used in the context of law education.

Re: AI outperforms law professors in Stanford Law study

#279

Post by scotty79 »

francisdavey wrote: post #267

I'm not a law lecturer. I spend most of my time wrangling contracts and advising about data law. But I did a stint of part-time work teaching a masters in law. My experience then (this was back before "Attention Is All You Need", I hadn't met the output of generative models) was that students tended to produce work that did not have a proper thread of reasoning in it. There was a tendency to repeat things they had re…

I'm curious what would be your take on the productions of this year's models.

Re: AI outperforms law professors in Stanford Law study

#280

Post by nickburns »

aristofun wrote: post #235

In general it is not surprising. Even if this particular study is bad. There are certain areas of law work that are about analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more. That is literally the bread of LLMs. Those types of lawyers should be the first in line for unemployment, not programmers, not even close.

'Bread *and butter'. The English expression requires the second part—but otherwise fits perfectly in your well-stated point, with which I wholeheartedly agree.

Source: AAL.

Post reply on HN