Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

arstechnica.com

Post reply on HN
Print view View original

91–100 of 212 posts

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#91

Post by lettergram »

travisjungroth wrote: post #87
cj wrote: post #19

I asked ChatGPT to tell me a riddle. It was “What is always hungry, needs to be fed, and makes your hands red?” (Or something like that) I asked for a hint about 5 times and it kept giving more legitimate sounding hints. Finally I gave up and asked for the answer to the riddle, and it spit out a random fruit which made no sense as the answer to the riddle. I then repeated the riddle and asked ChatGPT what the answer…

> That was a great way to show my non-tech family members the limitations of AI and why they shouldn’t trust it. These are the limitations of the version of ChatGPT you were using at that moment. They are not categorical limitations of AI or even LLMs. It’s amazing to me how many people are sleeping on AI, mixing up the failing cases of a freemium chatbot for the full capability of the tech, even on HN. LLMs can say…

I made https://AskHN.ai

What it does is not try to answer, but collect previous topics discussed by experts. Then answer the question based on the text, a far more reliable approach.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#92

Post by greenyoda »

rayiner wrote: post #17

Check out one of the fake opinions: https://storage.courtlistener.com/recap/gov.uscourts.nysd.57... . It even makes up a panel comprising real federal appellate judges (although one is from the fifth circuit while the fake case is from the eleventh circuit). I can see how someone unfamiliar with what GPT can do could get fooled.

This is interesting. At the beginning of the document, we read that Susan Varghese, representing the estate of George Varghese, "appeals the district court's dismissal of her wrongful death claim". However, in the "Factual background" section, we read about someone named Anish Varghese who is alleging a breach of contract due to being denied boarding on a flight and having to purchase another ticket.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#95

Post by okasaki »

gerdesj wrote: post #75
cj wrote: post #19

I asked ChatGPT to tell me a riddle. It was “What is always hungry, needs to be fed, and makes your hands red?” (Or something like that) I asked for a hint about 5 times and it kept giving more legitimate sounding hints. Finally I gave up and asked for the answer to the riddle, and it spit out a random fruit which made no sense as the answer to the riddle. I then repeated the riddle and asked ChatGPT what the answer…

> Or, learn how to say “I don’t know” It doesn't know that it doesn't know! It is, very roughly speaking, a model that is designed to print out the most likely word given its current input and training, and then the next word etc. Whereas you or I might be mistaken about some of our faculties, memories and skills, ChatGPT cannot possibly "know" what its limitations are. It was never taught what it was not taught (obv…

It seems that you don't know what you don't know, really.

There's no way to definitively know what properties ChatGPT has. It does seem to reason to some extent and it does often say that some information isn't known/there's no data. And it almost obnoxiously often tells you that it's simplifying a complex and multifaceted situation.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#96

Post by wintogreen74 »

jeroenhd wrote: post #31
lynx23 wrote: post #24

Wait, this is handled as "ChatGPT made up these cases" and not as "Layers deliberately used ChatGPT to fabricate stuff"? Is anyone really believing a lawyer is that stupid? I know, adssume good intentions and all, but in this case, really?

I believe some lawyers are that stupid. I also believe lawyers who are stupid enough not to verify ChatGPT's responses should be treated as if they willfully lied to the court. "Oops, I didn't know" is a good defence when you're caught accidentally walking on the grass, not when you're in court.

"Oops, I didn't know" has never been a valid defense for anyone; should definitely not be for lawyers, aka "people who are paid to know".

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#97

Post by Bilal_io »

travisjungroth wrote: post #87
cj wrote: post #19

I asked ChatGPT to tell me a riddle. It was “What is always hungry, needs to be fed, and makes your hands red?” (Or something like that) I asked for a hint about 5 times and it kept giving more legitimate sounding hints. Finally I gave up and asked for the answer to the riddle, and it spit out a random fruit which made no sense as the answer to the riddle. I then repeated the riddle and asked ChatGPT what the answer…

> That was a great way to show my non-tech family members the limitations of AI and why they shouldn’t trust it. These are the limitations of the version of ChatGPT you were using at that moment. They are not categorical limitations of AI or even LLMs. It’s amazing to me how many people are sleeping on AI, mixing up the failing cases of a freemium chatbot for the full capability of the tech, even on HN. LLMs can say…

> LLMs can say “I don’t know”. Even ChatGPT can do it.

That's the problem in my opinion. When you know something is capable of saying "I don't know" but confidently spits out some hallucinated BS is when the average person eats it up.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#98

Post by macjohnmcc »

cj wrote: post #19

I asked ChatGPT to tell me a riddle. It was “What is always hungry, needs to be fed, and makes your hands red?” (Or something like that) I asked for a hint about 5 times and it kept giving more legitimate sounding hints. Finally I gave up and asked for the answer to the riddle, and it spit out a random fruit which made no sense as the answer to the riddle. I then repeated the riddle and asked ChatGPT what the answer…

I have asked it to write code. The code looked reasonable but did not work (using a non-existent API or library). I said the library XXX doesn't exist. And it would respond with an apology and a new solution that maybe worked maybe didn't. Sometimes it could pull off a save eventually.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#99

Post by lamontcg »

lynx23 wrote: post #24

Wait, this is handled as "ChatGPT made up these cases" and not as "Layers deliberately used ChatGPT to fabricate stuff"? Is anyone really believing a lawyer is that stupid? I know, adssume good intentions and all, but in this case, really?

> Is anyone really believing a lawyer is that stupid?

There's over a million lawyers in the United States.

You'd expect at least one of them to be a 1-in-a-million level of bad, or 4.7 standard deviations below the mean assuming a Gaussian distribution of competency.

An average person would normally never come across that lawyer in their lifetime, but media will find that lawyer and amplify their mistakes to everyone in the population.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#100

Post by pavon »

lynx23 wrote: post #24

Wait, this is handled as "ChatGPT made up these cases" and not as "Layers deliberately used ChatGPT to fabricate stuff"? Is anyone really believing a lawyer is that stupid? I know, adssume good intentions and all, but in this case, really?

I can totally see a lawyer who hasn't been following closely read headlines about ChatGPT acing the Bar Exam, and start using it without being too skeptical.

What gets me is that they doubled down when asked to provide copies. Seriously, when that happens, you don't ask ChatGPT if the cases are real, you do your own damn search, and apologize profusely for your mistake. That really makes me question whether they were trying to pull a fast one, and then play dumb when caught, or if they really are that stupid.

Post reply on HN