Live data from Hacker News

Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

arstechnica.com

81–90 of 212 posts

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#81
post #69

How do these usually get checked? I mean, maybe he’s already done this successfully a few times.

They get checked by the lawyer for the other side.

Usually the first thing a Judge's clerk will do before a judge reads a brief is pull all of the authority cited for the judge to have ready-to-review along side the brief itself (if not analyzed, highlighted, summarized, etc. etc. depending on the judge).

Now, whether the judge actually reads them is debatable (I had my doubts sometimes). But you bet your ass that if the Clerk simply cannot find a case, the Judge will be informed of that.

YMMV in State courts, which can be all over the place in terms of professionalism. But you should at least assume your opponent is going to read your cases because the easiest way to beat someone in court is to point out the law you rely on is bad.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#82
post #14
post #10

It would have taken about 1 minute to put each of them into a tool like Casetext, Lexis, WL, or Bloomberg Law to determine they didn't exist.

To do that you first need to distrust AI, and a lot of people don't. They think of GPT like Google-but-written-in-English. That is a large part of the problem.

A lot of Google's top suggested results / autogenerated answers tell you nonsense or misunderstandings as well.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#83
post #24

Wait, this is handled as "ChatGPT made up these cases" and not as "Layers deliberately used ChatGPT to fabricate stuff"? Is anyone really believing a lawyer is that stupid? I know, adssume good intentions and all, but in this case, really?

I believe some lawyers are that stupid. I also believe lawyers who are stupid enough not to verify ChatGPT's responses should be treated as if they willfully lied to the court. "Oops, I didn't know" is a good defence when you're caught accidentally walking on the grass, not when you're in court.

This appears to be an "individual vs company" case, so I think expecting some cost-cutting on the individual's side is normal. Just not done this poorly.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#84
post #26
post #10

It would have taken about 1 minute to put each of them into a tool like Casetext, Lexis, WL, or Bloomberg Law to determine they didn't exist.

Exactly. This is how AI should be used. Maybe Chatgpt hallucinates a lot, but that doesn’t mean it’s useless. It can still provide a lot of good value, you just have to filter out the crap, just like you’d do with any other tool. Especially if you’re using it in this context, you should double check its facts. I’d say it’s more the fault of the lazy lawyer that didn’t bother to verify than the fault of ChatGPT.

100% agree.

I get a little annoyed at people seeing this AI, seeing how it's not absolutely perfect, and then acting like it's horrible.

I think the expression "All models are wrong but some are useful" applies very much to ChatGPT. It's a useful tool, even if it's not perfect.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#86
post #19

I asked ChatGPT to tell me a riddle. It was “What is always hungry, needs to be fed, and makes your hands red?” (Or something like that) I asked for a hint about 5 times and it kept giving more legitimate sounding hints. Finally I gave up and asked for the answer to the riddle, and it spit out a random fruit which made no sense as the answer to the riddle. I then repeated the riddle and asked ChatGPT what the answer…

Or, learn how to say “I don’t know”

It can't. Correct me if I'm wrong - I probably am - but as a generative language model I was under the impression it has no actual understanding of the meaning of it's words, only the probability they work well together.

You wouldnt expect dice to understand why you didn't want to roll snake eyes

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#87
post #19

I asked ChatGPT to tell me a riddle. It was “What is always hungry, needs to be fed, and makes your hands red?” (Or something like that) I asked for a hint about 5 times and it kept giving more legitimate sounding hints. Finally I gave up and asked for the answer to the riddle, and it spit out a random fruit which made no sense as the answer to the riddle. I then repeated the riddle and asked ChatGPT what the answer…

> That was a great way to show my non-tech family members the limitations of AI and why they shouldn’t trust it.

These are the limitations of the version of ChatGPT you were using at that moment. They are not categorical limitations of AI or even LLMs.

It’s amazing to me how many people are sleeping on AI, mixing up the failing cases of a freemium chatbot for the full capability of the tech, even on HN. LLMs can say “I don’t know”. Even ChatGPT can do it. Ask some super niche historical questions of any version and see what you get. Is it perfect every time? No. But that’s something that can be reduced.

Over the next year, you’ll see more instances of lawyers citing hallucinated cases. There will also be a handful of startups that hook up LLMs to document stores, and they’ll be able to check for this sort of thing and do an even better job.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#88
post #46

For reference, here is the judge’s order. Endnote #2 alone is worth the price of admission: https://s3.documentcloud.org/documents/23826753/judgeaskingt...

That guy has some work to do on his resume.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#89

Sigh, I'm getting very sick of hearing about how "ChatGPT" makes stuff up. Yes, 3.5 made a lot of stuff up, 4.0 still does, but it's much rarer. I wish people would mention this, it's all treated as the same thing. It's like talking about how unreliable these "Airplanes" are when they are talking about prop planes, even though jets are out.

The difference is vast and somewhat irritating to those with GPT-4 access today, but it’s also besides the point. The only people who deserve to cringe at the conflation are OpenAI’s marketing team, but even they may take solace in the sheer cut-through of column inches their cloven brand obtains, no doubt anticipating a future retirement of GPT-3.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#90
post #19

I asked ChatGPT to tell me a riddle. It was “What is always hungry, needs to be fed, and makes your hands red?” (Or something like that) I asked for a hint about 5 times and it kept giving more legitimate sounding hints. Finally I gave up and asked for the answer to the riddle, and it spit out a random fruit which made no sense as the answer to the riddle. I then repeated the riddle and asked ChatGPT what the answer…

I've had a few moments with ChatGPT that are great anecdotes similar to your own: - Asked it to generate a MadLib for me to play that was no more than a paragraph long. It produced something that was several paragraphs wrong. I told it "no. That's X paragraphs. I asked for one that is only 1 paragraph long" and it would respond "I'm sorry for the misunderstanding. Let me try again" and then would make the same mistak…

Did you really expect ChatGPT to be a full blown dungeon master?
Post reply on HN