Live data from Hacker News

AI outperforms law professors in Stanford Law study

law.stanford.edu

351–360 of 384 posts

Re: AI outperforms law professors in Stanford Law study

#351
Law and accounting both seem to be the perfect fields to replace with AI.

Just massive data where you either do calculations or interpretation.

You will replace 100 lawyers with AI and have a single lawyer to review what the AI outputs and stamp their name on it for accountability.

Re: AI outperforms law professors in Stanford Law study

#352
post #234

Earlier quoted context omitted.

> Just a humble opinion that I would love to see be wrong Out of curiosity, why would you love to be wrong about that? What possible outcome could you see being a net positive for society if the vast majority of knowledge workers (and ultimately, as robotics progress, most workers in general) are replaced by AI?

I believe it was Blink-182 who said, "Work sucks". You have to pay people to do that stuff; they don't want to be there. And then you get into second order effects- costs plummet for anything labor intensive, including medical care, prepared food, cleaning, and private tutors. Then onto tertiary effects- if you can spin up a million genius researchers to attack a problem, you start seeing massive progress in every im…

I think a lot of the time when this debate occurs (which, at this point, literally every single day I see something about this) UBI is almost always the contention point but I feel like that's really not the end-all... Like, sure, say there's a miracle and we have UBI get instituted. I think that is maybe 25% of the solution. The other problem is now you're going to have an entire class of people basically living without a purpose. Yeah, I get it, they can go and "explore their passions" and focus on "creative works" or whatever BS people persuade themselves into thinking the vast majority of society would want to do, but realistically I think there would be a huge psychological breakdown in people now living without a fundamental purpose in society.

Somewhat related to that -- I was just this weekend watching a YouTube essay about PTSD in knights back in the medieval times, and the main point made in the video is that the psychological impacts incurred by the knights after battle were not just from seeing fucked up shit... the most apparent and serious cases of "PTSD" occurred when a knight was injured enough on the battle field resulting in them no longer able to be soldiers. Their entire purpose in the world got stripped away resulting in serious psychological stress. I think that same issue would apply to many people today (lawyers, engineers, investment bankers, etc) who would no longer be able to practice their craft. (This is the video for reference, was a good watch https://www.youtube.com/watch?v=849dmdc-Qf8)

I understand the counter argument to this is going to be some anti-capitalist rhetoric like "Well people shouldn't live to be workers and that's fucked up that they have live that way!" but IMO, some people like what they do and don't want to be made useless. (Not implying that is what you were insinuating, but just in a broad sense I that genera of argument doesn't make sense to me)

Re: AI outperforms law professors in Stanford Law study

#353
post #7

As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.

There will still need to be a lawyer in the loop to review and stamp and take accountability.

However, the good news is that a whole bunch of laywer positions in drafting docs and research will be able to be eliminated due to AI.

Re: AI outperforms law professors in Stanford Law study

#354
post #198

Earlier quoted context omitted.

The issue is, it almost always outperforms knowledge workers. IF the right questions are asked, and IF steered into and corrected at a few crucial points. IF not it goes off in the wrong direction really quick and that's a problem that's still mostly unsolved in the last 2 years. And that can be catastrophic in high risk environments, like legal, medical or high risk software products where being wrong in the wrong p…

Ya, while the tools are really solid and have seen huge leaps these past two years, in no way will an LLM be able to do any of it unguided in two years. Just a humble opinion that I would love to see be wrong.

There's a huge difference between one shot and few shot versus building a robust harness with deterministic and adversarial quality gates. And I'm finding that agents can actually do a pretty good job of a surprising number of things if you are very clear about your dimensions of quality and the rubrics that you get agents to research and then use to validate against those dimensions of quality.

Make sure to use a deterministic pipeline or harness to go step by step so agents aren't checking their own work and I sometimes get alpha from having a codex check the work of a clod but I am seeing pretty good output across multiple domains when I have three independent quality gates and a loop which only spits it out to a human if it doesn't converge at a reasonable cost.

Re: AI outperforms law professors in Stanford Law study

#355

I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's als…

More than that, the entire structure of the study is pointless. They set up as a question/response and then had humans rate the response. That's literally what LLM's are trained to do, which ultimately is convincing a human to click the "I like this one better" button on it's response.

This is kind of like saying you can't compare Computer Vision models to Human performance because those models were literally trained to identify objects in images...

Re: AI outperforms law professors in Stanford Law study

#356

Earlier quoted context omitted.

The issue is that before GPT models basically were useless for any conversation. We are literally in science fiction realm. From a text conversation perspective the gap between where we are at and what’s left to get to is relatively small. In my opinion, the main thing we need to do is have training happen continuously. And probably more real world data (from sensors).

The ELIZA effect has been around since 1966. I think lots of folks feel “AI” has advanced much more quickly that it really has because of the nature of its many past boom / bust cycles.

ELIZA has never done well in conversational tests. GPT-4.5, for example, tends to out-human humans. Like I could never ask ELIZA this question and get anything close to a decent response: "Give me three points that convey the impact that 9/11 had on rap music in the 21st century with some good examples?" Asking ChatGPT today gives me an answer that I'd give an A grade to a strong college student. ELIZA's response -- "What do you think?".

Re: AI outperforms law professors in Stanford Law study

#357
post #234

Earlier quoted context omitted.

> Just a humble opinion that I would love to see be wrong Out of curiosity, why would you love to be wrong about that? What possible outcome could you see being a net positive for society if the vast majority of knowledge workers (and ultimately, as robotics progress, most workers in general) are replaced by AI?

I believe it was Blink-182 who said, "Work sucks". You have to pay people to do that stuff; they don't want to be there. And then you get into second order effects- costs plummet for anything labor intensive, including medical care, prepared food, cleaning, and private tutors. Then onto tertiary effects- if you can spin up a million genius researchers to attack a problem, you start seeing massive progress in every im…

> I believe it was Blink-182 who said, "Work sucks". You have to pay people to do that stuff; they don't want to be there.

Believe it or not, some people actually do enjoy their jobs and work they do.

> I get that you might have a 'UBI/alternative general welfare is impossible' up your sleeve, but you've written this like it's somehow unfathomable that not forcing everybody to work just to survive would be a good thing.

UBI absolutely is unfathomable here (US). The USG won't even give people health care. People go bankrupt to afford life saving care on a regular basis. Or just die... Even if those cases are a minority, just the fact that it happens says a lot. So I do think it is unfathomable that UBI would be implemented here. I don't think that's unreasonable to say.

Re: AI outperforms law professors in Stanford Law study

#358

Earlier quoted context omitted.

Nobody is disputing that. I specifically said that I can see the improvements from the last six months. What I’m saying is we can’t assume that every two years it will improve at the same rate. The further we get into this, the more AI feels like 3-D printing. Significantly bigger and will be more widely used for sure. But nowhere near the “new industrial revolution” that all these companies are making it out to be

What would 3D printing have to do in order for it to be the new industrial revolution to you?

Everyone has one at home spitting out items they need daily/weekly like was promised. I don't know if you remember the 3D printer (somewhat) boom of the 2010's but the hype was crazy when it became more mainstream. Maker spaces popping up in cities everywhere, schools showing off their units, every conference had some talk on them, startups left and right. The AI boom is basically a more-funded version of that. It was hot hot hot and people thought every home was bound to have one.

Re: AI outperforms law professors in Stanford Law study

#359

Earlier quoted context omitted.

more and more i see papers. interview 8 ppl, draw conclusions based on their expert opinions. AI and Cybersecurity are full of this. Even saw some where they just slapped interviews + protocol into chatgpt as 'methodology' to extract the results -_-. Peer reviewed and published.

People don't always have the resources to conduct massive "proper" studies. We live in the real world, and have to settle for what studies people can conduct. Not saying we should take such studies as the "gospel truth" ... but if you ignore them and only consider "proper" studies, you'll be waiting a very long time to learn anything new.

You are saying the companies that are planning to build structures the size of Manhattan, while claiming multiple trillions TAM, and eventual apotheosis, along with the consumers of these models can't scrape together enough coins to fund a study with a decent statistical power?

We have to settle for 'crumbs'?

Why would you say this like it is true?

Re: AI outperforms law professors in Stanford Law study

#360

Earlier quoted context omitted.

Assuming it keeps improving at the same rate, which I think we are already seeing not play out. If you compare the first six months when GPT truly hit the mainstream to the previous six months, the improvements are not nearly as evident. That isn’t to say they aren’t noticeable, I could definitely tell it’s improving, but not nearly at the pace it once was. There’s also the fact that they can’t possibly keep improvin…

The issue is that before GPT models basically were useless for any conversation. We are literally in science fiction realm. From a text conversation perspective the gap between where we are at and what’s left to get to is relatively small. In my opinion, the main thing we need to do is have training happen continuously. And probably more real world data (from sensors).

> what’s left to get to is relatively small

Not necessarily. In many (most?) areas of tech the rate of advancement follows a logarithmic curve. That is to say, the first 90% is achieved quickly but the last 10% takes significantly more time.

Post reply on HN