Live data from Hacker News

AI outperforms law professors in Stanford Law study

law.stanford.edu

301–310 of 384 posts

Re: AI outperforms law professors in Stanford Law study

#301

Earlier quoted context omitted.

'Bread *and butter'. The English expression requires the second part—but otherwise fits perfectly in your well-stated point, with which I wholeheartedly agree. Source: AAL.

Thank you! As a non native speaker I was not sure if “and butter” is a mandatory part but didn’t want (nor had time) to llm the comment for the sake of authenticity :) TIL

Ah, great! I 'took a shot in the dark' there. Glad it 'panned out'. ;)

Re: AI outperforms law professors in Stanford Law study

#302

Earlier quoted context omitted.

On the other hand we keep seeing only marginal generational imorovements in CPU space, yet performance gains over last 10 years in CPUs are very material. Every new model might not be a leap like it used to be, but give it enough time and improvements add up.

Nobody is disputing that. I specifically said that I can see the improvements from the last six months. What I’m saying is we can’t assume that every two years it will improve at the same rate. The further we get into this, the more AI feels like 3-D printing. Significantly bigger and will be more widely used for sure. But nowhere near the “new industrial revolution” that all these companies are making it out to be

What would 3D printing have to do in order for it to be the new industrial revolution to you?

Re: AI outperforms law professors in Stanford Law study

#303

I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's als…

Reversly viewed ones should ask with what intend the study should be like this. And for obvious reasons it sounds like monetary-nature.

Re: AI outperforms law professors in Stanford Law study

#304

I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's als…

Sure, but in two years AI has gone from “impressive tool, but not a replacement for knowledge workers” to “the study where it beats our highest caliber of knowledge workers may have some methodological deficits.” In another two years it’s going to be curtains.

I will never trust an AI as much as a person

Re: AI outperforms law professors in Stanford Law study

#305

I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's als…

More than that, the entire structure of the study is pointless. They set up as a question/response and then had humans rate the response. That's literally what LLM's are trained to do, which ultimately is convincing a human to click the "I like this one better" button on it's response.

Re: AI outperforms law professors in Stanford Law study

#306

Earlier quoted context omitted.

I kinda disagree. High risk environments just means that they will have to have a human-in-the-loop for a longer time which drastically reduce the skill required for such human (which is still requires high skill just not stupidly high).

> drastically reduce the skill required for such human I mean thats what is wanted by some companies. The problem, especially for things like legal is that it requires someone more skilled to read through and understand that the argument is bollocks, or the law/precedent they are banking on is in fact the right one. We have a tool that auto-writes letters to our management companies when they break SLAs. We have a sl…

Man, this comment made me think of a Kafkaesque future where two AI lawyers and an AI Judge are stuck in an infinite loop arguing over a case, meanwhile the defendant is running around trying to get anyone in the legal system to recognize that the AI is stuck.

Re: AI outperforms law professors in Stanford Law study

#307
post #234

Earlier quoted context omitted.

> Just a humble opinion that I would love to see be wrong Out of curiosity, why would you love to be wrong about that? What possible outcome could you see being a net positive for society if the vast majority of knowledge workers (and ultimately, as robotics progress, most workers in general) are replaced by AI?

I believe it was Blink-182 who said, "Work sucks". You have to pay people to do that stuff; they don't want to be there. And then you get into second order effects- costs plummet for anything labor intensive, including medical care, prepared food, cleaning, and private tutors. Then onto tertiary effects- if you can spin up a million genius researchers to attack a problem, you start seeing massive progress in every im…

In that scenario, AI would have to be a public utility, which it is not. Private corporations have no intention to provide services for public good. If they displace a billion jobs, they'll just throw up their hands and go "we're just an Ai company guyz"

Re: AI outperforms law professors in Stanford Law study

#308

I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's als…

> There's also really clear bias given that the main results only feature Google models.

The main results also don’t seem to know what a “model” is, as the two “models” it refers to are “stock Gemini 2.5 Pro” and “a retrieval-augmented version of NotebookLM”.

One of which is a model, and the other of which is an interface backed by different models depending on exactly when the analysis was performed.

Re: AI outperforms law professors in Stanford Law study

#309

In general it is not surprising. Even if this particular study is bad. There are certain areas of law work that are about analyzing large amounts of texts, drawing conclusions and writing other texts based on that and nothing more. That is literally the bread of LLMs. Those types of lawyers should be the first in line for unemployment, not programmers, not even close.

I see the same problem with AI in both programming and law though. AI is like a scab on a wound: it's a temporary filler, it rushes in to fill a void, but it's not going to be the final solution. Models showed us that there was huuuge unmet demand for literacy, both in software and in law. But now we have a choice to either address the systemic causes of the unmet demand, or just try to paper over them with layers an…

> But now we have a choice to either address the systemic causes of the unmet demand, or just try to paper over them with layers and layers of AI scab.

Yeah, but in my experience it won't come down to "which is the better solution" but "which is cheaper/easier"

So I look forward to lots of layers of papered over AI scabs in the future. It won't be cheaper in the long run, but it will pump someone's quarterly numbers enough that they get a promotion before the problem they introduce come back to them

Re: AI outperforms law professors in Stanford Law study

#310

I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's als…

Sure, but in two years AI has gone from “impressive tool, but not a replacement for knowledge workers” to “the study where it beats our highest caliber of knowledge workers may have some methodological deficits.” In another two years it’s going to be curtains.

Autopilots have been able to land planes for years (decades?), and yet they still don't land passengers planes at any increased rate.
Post reply on HN