Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI outperforms law professors in Stanford Law study

law.stanford.edu

Post reply on HN
Print view View original

321–330 of 384 posts

Re: AI outperforms law professors in Stanford Law study

#321

Post by hungryhobbit »

saidnooneever wrote: post #318
godelski wrote: post #101

I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's als…

more and more i see papers. interview 8 ppl, draw conclusions based on their expert opinions. AI and Cybersecurity are full of this. Even saw some where they just slapped interviews + protocol into chatgpt as 'methodology' to extract the results -_-. Peer reviewed and published.

People don't always have the resources to conduct massive "proper" studies. We live in the real world, and have to settle for what studies people can conduct.

Not saying we should take such studies as the "gospel truth" ... but if you ignore them and only consider "proper" studies, you'll be waiting a very long time to learn anything new.

Re: AI outperforms law professors in Stanford Law study

#322

Post by [deleted] »

ryan_n wrote: post #234
goolz wrote: post #198

Earlier quoted context omitted.

Ya, while the tools are really solid and have seen huge leaps these past two years, in no way will an LLM be able to do any of it unguided in two years. Just a humble opinion that I would love to see be wrong.

> Just a humble opinion that I would love to see be wrong Out of curiosity, why would you love to be wrong about that? What possible outcome could you see being a net positive for society if the vast majority of knowledge workers (and ultimately, as robotics progress, most workers in general) are replaced by AI?

[deleted]

Re: AI outperforms law professors in Stanford Law study

#323

Post by bookofjoe »

ToValueFunfetti wrote: post #297
ryan_n wrote: post #234

Earlier quoted context omitted.

> Just a humble opinion that I would love to see be wrong Out of curiosity, why would you love to be wrong about that? What possible outcome could you see being a net positive for society if the vast majority of knowledge workers (and ultimately, as robotics progress, most workers in general) are replaced by AI?

I believe it was Blink-182 who said, "Work sucks". You have to pay people to do that stuff; they don't want to be there. And then you get into second order effects- costs plummet for anything labor intensive, including medical care, prepared food, cleaning, and private tutors. Then onto tertiary effects- if you can spin up a million genius researchers to attack a problem, you start seeing massive progress in every im…

"Work is what you're doing when you'd rather be doing something else."

Re: AI outperforms law professors in Stanford Law study

#324

Post by dcre »

volkercraig wrote: post #305
godelski wrote: post #101

I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's als…

More than that, the entire structure of the study is pointless. They set up as a question/response and then had humans rate the response. That's literally what LLM's are trained to do, which ultimately is convincing a human to click the "I like this one better" button on it's response.

They're only good at it because that's what they're good at? Come on.

Re: AI outperforms law professors in Stanford Law study

#325

Post by vonunov »

jvvw wrote: post #294
pezgrande wrote: post #227

Earlier quoted context omitted.

I kinda disagree. High risk environments just means that they will have to have a human-in-the-loop for a longer time which drastically reduce the skill required for such human (which is still requires high skill just not stupidly high).

The employers will think it requires less skill, whereas in fact it might actually require more skill to do a good job of being the human-in-the-loop. For example, my sister is a translator and she says that checking AI translations is actually harder in many ways than doing a translation in the first place, but the agencies pay less for checking than actual translation.

I used to do audio transcription and some video captioning. Found it a bit drudgerous and fatiguing in rather specific ways, but I was effective at it and could find some satisfaction in it. It's been some years now, so I haven't had a chance to try out the kind of thing they're doing now, but I'm pretty sure I wouldn't want to. I can raise my blood pressure just sitting here and thinking about what it would be like to have to go through a Word doc and correct the bot's errors. But, even putting aside my professional pride (or indignation), I can only imagine that it would make all kinds of mistakes I never would, and wouldn't be any help with the parts I'd have trouble with. And I'm pretty sure that, at least often enough for it to be an issue, the priming of reading what the bot thought something was could easily make it way harder to hear it correctly, if I notice there's something wrong in the first place. I assume there's a similar problem for your sister along the lines of throwing off how it would occur to her to express something in the target language.

Re: AI outperforms law professors in Stanford Law study

#326

Post by runarberg »

pezgrande wrote: post #231
runarberg wrote: post #125

Earlier quoted context omitted.

The study was conducted by Stanford’s HAI institute, which receives heavy funding from Google (how much I couldn’t find because they don‘t publish their donations in a place I could find it; but I suspect it is alot). And the authors did not declare a non-conflict of interest at the end of the paper.

Do papers need a "non-conflict of interest" disclosure nowadays to not be considered just ads?

When they are studying a consumer product it is pretty customary to declare a non-conflict of interest. So yes. Declare it at the end of your paper please.

Unless you have a conflict of interest, in which case declare e.g. “the lead author of this paper is an Associate Director and Senior Fellow at HAI which receives funding from Google the company which makes Gemini”.

Re: AI outperforms law professors in Stanford Law study

#327

Post by bluefirebrand »

godelski wrote: post #85
thewebguyd wrote: post #15

Earlier quoted context omitted.

I think this is probably true for most skilled professions. AI is best used in the hands of folks already knowledgeable in the skills/professions they are using it for. I liken it to me googling things as a sysadmin vs. Jane from accounting doing it. The non-tech end user is far more likely to make the problem worse, or install something sketchy from the ad riddled results than I am, or one of my help desk employees…

> I think this is probably true for most skilled professions. I agree, BUT I also find that it's easy for experts to atrophy quickly. When the AI is right 80/90% of the time it lulls you into over confidence. I find those that are best and make the greatest use are the ones who remain skeptical but also use the tool. The same people who were already nuanced and picky before AI. The same people who already doubted and…

> I agree, BUT I also find that it's easy for experts to atrophy quickly. When the AI is right 80/90% of the time it lulls you into over confidence

Thinking the AI is right 80/90% of the time is already a sign of being lulled into overconfidence. The actual percentage is much lower in my experience. I'm willing to grant the AI is "somewhat right" that often but is that really what we settle for?

Am I secretly the only person who ever actually cared about being very accurate. Is AI just an excuse everyone else is using so they can stop pretending? This is so incredibly frustrating

> If you weren't willing to just "lgtm" with your own code, it's difficult to do that with AI.

If you are willing to do that with your own code you should probably not be trusted to work on software

Re: AI outperforms law professors in Stanford Law study

#328

Post by bluefirebrand »

ChrisMarshallNY wrote: post #268
tiahura wrote: post #259

Earlier quoted context omitted.

Corporations are DEMANDING legal ai because it is so much more efficient.

Lawyers creating legal stuff, via LLMs is OK. Programmers creating software through LLMs is OK. Mixing them, is, not, in my experience, OK. In the future, I am sure that LLMs will reach the point, where their output will be beyond reproach, but we're not there, yet. That means that someone that knows the context and content, needs to vet the output, before sending it on.

> In the future, I am sure that LLMs will reach the point, where their output will be beyond reproach, but we're not there, yet.

I have no doubt that you're right, but will it be because they are close to infallible or because we have let ourselves become lazy and reliant?

My money is on lazy and reliant based on the trends I'm actually seeing

Re: AI outperforms law professors in Stanford Law study

#329

Post by FromTheFirstIn »

dcre wrote: post #324
volkercraig wrote: post #305

Earlier quoted context omitted.

More than that, the entire structure of the study is pointless. They set up as a question/response and then had humans rate the response. That's literally what LLM's are trained to do, which ultimately is convincing a human to click the "I like this one better" button on it's response.

They're only good at it because that's what they're good at? Come on.

They’re not good at it because they understand the law

Re: AI outperforms law professors in Stanford Law study

#330

Post by runarberg »

hungryhobbit wrote: post #321
saidnooneever wrote: post #318

Earlier quoted context omitted.

more and more i see papers. interview 8 ppl, draw conclusions based on their expert opinions. AI and Cybersecurity are full of this. Even saw some where they just slapped interviews + protocol into chatgpt as 'methodology' to extract the results -_-. Peer reviewed and published.

People don't always have the resources to conduct massive "proper" studies. We live in the real world, and have to settle for what studies people can conduct. Not saying we should take such studies as the "gospel truth" ... but if you ignore them and only consider "proper" studies, you'll be waiting a very long time to learn anything new.

These studies are often conducted by the AI companies them selves (in this case, an institute that receives funding from AI companies), if they were interested in the truth (which they obviously are not) and not propaganda (which they obviously are) they would fund the necessary research. AI companies have plenty of money and can well afford to do this properly.

Other then AI companies, a more realistic option are state funded universities (particularly in Europe and east Asia) which have consumer protection agencies who’s purpose is to protect their residents from corporate greed, and as such should fund, commission, or even conduct such studies. They also have enough money to do this properly.

If there is enough money for propaganda, there should also be enough money for the truth.

Post reply on HN