Live data from Hacker News

AI outperforms law professors in Stanford Law study

law.stanford.edu

11–20 of 384 posts

Re: AI outperforms law professors in Stanford Law study

#11
post #9
post #5

[flagged]

Just so you know, I have nothing to do with Stanford, but I am flagging this as conspiratorial nonsense. So when you comment is flagged, I just want you to know that it doesn't confirm your belief, it's just that this comment harms discussion and so must be removed.

>Don't feed egregious comments by replying; flag them instead. If you flag, please don't also comment that you did.

https://news.ycombinator.com/newsguidelines.html

Re: AI outperforms law professors in Stanford Law study

#13
post #10
post #6

> In a blind evaluation of nearly 3,000 anonymized comparisons, professors rated AI responses significantly higher than answers written by other professors, with AI winning 75% of head-to-head matchups. 75% win rate seems pretty good! Paper link: https://law.stanford.edu/wp-content/uploads/2026/06/salinas_...

I do wish they'd used some more objective criteria. Simply being preferable one of the things LLMs have trained for since the beginning, hence its sycophantic nature.

What criteria would you use for judging legal arguments?

Re: AI outperforms law professors in Stanford Law study

#15
post #7

As a software engineer I have some intuition for what the risks are of letting agents do some tasks vs others. I don't have a similar intuition calibrated for what could go wrong when asking AI to draft a legal document. Some things seem harmless, i.e. drafting a will, but I don't really know- our legal system is notoriously rife with footguns.

I think this is probably true for most skilled professions. AI is best used in the hands of folks already knowledgeable in the skills/professions they are using it for.

I liken it to me googling things as a sysadmin vs. Jane from accounting doing it. The non-tech end user is far more likely to make the problem worse, or install something sketchy from the ad riddled results than I am, or one of my help desk employees are.

I wouldn't trust myself to draft an important legal document using AI without the advice of a lawyer, much like I wouldn't really want to rely on my lawyer to use AI to write code for me.

Re: AI outperforms law professors in Stanford Law study

#16
post #6

> In a blind evaluation of nearly 3,000 anonymized comparisons, professors rated AI responses significantly higher than answers written by other professors, with AI winning 75% of head-to-head matchups. 75% win rate seems pretty good! Paper link: https://law.stanford.edu/wp-content/uploads/2026/06/salinas_...

Yeah, 75% win rate is a ~200 points Elo difference, which is quite massive.

Re: AI outperforms law professors in Stanford Law study

#20
post #3

I think there will be a market for firms that aggressively market themselves as non-AI, and then as more people turn towards that human connection we'll go full circle

If you want human connection the legal system is not where you are going to find it, period. I don't think there will be any such market for "non ai" law. If I'm involved with the legal system I just want out as quick as possible as cheap as possible.

Bad legal advice will keep you dealing with the legal system for much longer and at much greater cost. Something being cheap and quick upfront doesn't mean it will be cheap and quick by the end of the process.
Post reply on HN