Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI outperforms law professors in Stanford Law study

law.stanford.edu

Post reply on HN
Print view View original

121–130 of 384 posts

Re: AI outperforms law professors in Stanford Law study

#121

Post by epicureanideal »

One way to make legal services more affordable and accessible would be to put the burden of ensuring the AI legal services are accurate on a private-public partnership with the government.

If a person using the service is given inaccurate legal advice and acts on that advice, the person can't be charged with a crime, can't be given any civil penalties, etc., as long as the law in question is non-obvious.

Obviously if by some exploit, some fundamentally obvious crime (murder, theft, obvious fraud, etc.) is said to be legal, that wouldn't apply, but of course the service should try to prevent those kinds of exploits anyway.

Could limit this to something like business regulations to begin with, or even specifically for small businesses, or contracts within some time limit and dollar amount that would otherwise be coverable by small claims court, etc.

Re: AI outperforms law professors in Stanford Law study

#122

Post by b40d-48b2-979e »

Merad wrote: post #118
causal wrote: post #7

As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.

> Some things seem harmless, i.e. drafting a will Absolutely not harmless if you're the executor of an estate forced to deal with a screwed up AI will. I just handler my dad's estate this spring. It's a frustrating and confusing process even with the simplest of estates.

Most people don't have anything that could even be called an "estate".

Re: AI outperforms law professors in Stanford Law study

#123

Post by rockskon »

I do question at what point AI could be useful as a teaching aid.

The quality of LLMs depends heavily on, among other things, how you word your questions.

Knowing the correct questions to ask is not something most students know how to do given that it tends to require a fair bit of pre-existing domain knowledge.

Re: AI outperforms law professors in Stanford Law study

#124

Post by nocoiner »

b40d-48b2-979e wrote: post #122
Merad wrote: post #118

Earlier quoted context omitted.

> Some things seem harmless, i.e. drafting a will Absolutely not harmless if you're the executor of an estate forced to deal with a screwed up AI will. I just handler my dad's estate this spring. It's a frustrating and confusing process even with the simplest of estates.

Most people don't have anything that could even be called an "estate".

Everyone has an estate. Only thing is that you have to die first.

Re: AI outperforms law professors in Stanford Law study

#125

Post by runarberg »

godelski wrote: post #101

I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's als…

The study was conducted by Stanford’s HAI institute, which receives heavy funding from Google (how much I couldn’t find because they don‘t publish their donations in a place I could find it; but I suspect it is alot). And the authors did not declare a non-conflict of interest at the end of the paper.

Re: AI outperforms law professors in Stanford Law study

#126

Post by mdlman »

U4E4 wrote: post #116
applicative wrote: post #93

What the LLM cannot do is explain why it said what it said, when cross-examined. It simply hallucinates the best account of why someone would have said such a thing as it said, same as it can give a probable account of why someone else said something different. The question 'But why did you say this not that ...?' does not lead it to make explicit its grounds for what it said, but just to make a new more complicated…

This is true in the naive case. There are however LLM context building techniques that anchor completions in data structures that persist the structure of claims that support the conclusion contained in a completion. Lots of different patterns exist —organizing logic in language is a rich domain— but the one I’ve liked the most is something called a Claim Dependency Graph that models the relationships between atomic…

I’d love to read more about these type of patterns. Do you have any recommendations?

Re: AI outperforms law professors in Stanford Law study

#127

Post by acdha »

b40d-48b2-979e wrote: post #122
Merad wrote: post #118

Earlier quoted context omitted.

> Some things seem harmless, i.e. drafting a will Absolutely not harmless if you're the executor of an estate forced to deal with a screwed up AI will. I just handler my dad's estate this spring. It's a frustrating and confusing process even with the simplest of estates.

Most people don't have anything that could even be called an "estate".

It’s just the legal term. If you have a relative die with a bit of stuff and an ancient car, they have an estate and someone needs to execute it even if the total value is less than most lawyers care about.

Re: AI outperforms law professors in Stanford Law study

#128

Post by ALittleLight »

godelski wrote: post #101

I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's als…

The paper says the professors have a median of 200 comparisons each. It also says they only used 2 models because using more models would require more comparisons and they selected Google models because Google was branded/advertised as being education focused. When you see other models show up elsewhere, that's because they extended the main idea to other models but using LLMs to judge instead of human professors.

Re: AI outperforms law professors in Stanford Law study

#129

Post by godelski »

ALittleLight wrote: post #128
godelski wrote: post #101

I find this study quite suspect. I'd have to dive deeper but there's definitely significant alarm bells that should be going off for anyone reading. Figure 2 (page 6) screams problems. There's only 16 professors (3k comparisons each?!?!) and the professors are all over the place. That's very high variance, suggesting the study has no meaningful statistical power. Poor instructor 16 can't catch a break lol There's als…

The paper says the professors have a median of 200 comparisons each. It also says they only used 2 models because using more models would require more comparisons and they selected Google models because Google was branded/advertised as being education focused. When you see other models show up elsewhere, that's because they extended the main idea to other models but using LLMs to judge instead of human professors.

Sure, but the biggest problem is they have no statistical significance. Variance is too high. How do you distinguish the signal from the noise? Confidence intervals aren't enough.

But is it a surprise law professors aren't great statisticians?

Re: AI outperforms law professors in Stanford Law study

#130

Post by jcranmer »

b40d-48b2-979e wrote: post #122
Merad wrote: post #118

Earlier quoted context omitted.

> Some things seem harmless, i.e. drafting a will Absolutely not harmless if you're the executor of an estate forced to deal with a screwed up AI will. I just handler my dad's estate this spring. It's a frustrating and confusing process even with the simplest of estates.

Most people don't have anything that could even be called an "estate".

Judging from reported figures, roughly 80-90% of households in the US [1] have a household net worth of at least $0. That means that most people do in fact have an estate.

Median household net worth is in fact somewhere in the $100k-200k range, which is definitely something that could be meaningfully called an "estate." (Most of this tends to be the house, the median net equity in which is about $190k as of 2022).

Source: https://www2.census.gov/library/publications/2024/demo/p70br...

[1] This doesn't mean "homeowners," rather it's a recognition that assets for married or cohabitating couples are usually commingled.

Post reply on HN