Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lawyer cites fake cases invented by ChatGPT, judge is not amused

simonwillison.net

Post reply on HN
Print view View original

51–60 of 319 posts

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#51

Post by metadat »

_Microft wrote: post #32
metadat wrote: post #8

Earlier quoted context omitted.

Thanks! Macro-expanded: ChatGPT-Authored Legal Filing “Replete with Citations to Non-Existent Cases" https://news.ycombinator.com/item?id=36092509 (67 comments, 1 day ago)

dang is doing that frequently - but what does "macro-expanded" mean for normal users? Do you have a bookmarklet, extension or something similar for this or is there any secret incantation that makes HN generate such an entry automatically?

I just do it manually.. I've gotten pretty efficient at it, takes only a moment.

Imo including title, timeline / age, and comment volume provides helpful context to readers (I always appreciate when others do this, rather than, in the most severe cases, leaving a wall of unadorned HN links).

Cheers _Microft (and cool username, btw ;D)

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#52

Post by acka »

grumple wrote: post #40
taberiand wrote: post #28

Earlier quoted context omitted.

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

"It doesn't lie, it just generates lies and printed them to the screen!" I don't think there's a difference.

Does a piece of software with a bug in it which causes it to produce incorrect output lie or is it simply a programming error? Did the programmer who wrote the buggy code lie? I don't think so.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#53

Post by jimsimmons »

akiselev wrote: post #39

I went ahead and asked ChatGPT with the browsing plugin [1] because I was curious and it answered that it was a real case citing an article about the fake citations! After some prodding ("Are you sure?") it spat out something slightly saner citing this very article! > The case "Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019)" was cited in court documents, but it appears that there might…

In the loop there indeed was a allegedly trained human in this instance

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#54

Post by mod50ack »

grumple wrote: post #40
taberiand wrote: post #28

Earlier quoted context omitted.

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

"It doesn't lie, it just generates lies and printed them to the screen!" I don't think there's a difference.

Lying implies an intention. ChatGPT doesn't have that.

What ChatGPT definitely does do is generate falsehoods. It's a bullshitting machine. Sometimes the bullshit produces true responses. But ChatGPT has no epistemological basis for knowing truths; it just is trained to say stuff.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#56

Post by zdragnar »

grumple wrote: post #40
taberiand wrote: post #28

Earlier quoted context omitted.

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

"It doesn't lie, it just generates lies and printed them to the screen!" I don't think there's a difference.

The difference is everything. It doesn't understand intent, it doesn't have a motivation. This is no different than what fiction authors, songwriters, poets and painters do.

The fact that people assume what it produces must always be real because it is sometimes real is not its fault. That lies with the people who uncritically accept what they are told.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#57

Post by fortyseven »

simonw wrote: post #14
mehwoot wrote: post #7

No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? It did this because it's copying how humans talk, not what humans do. Humans say "I double checked" when asked to verify so…

Yeah, that was my conclusion too: What’s a common response to the question “are you sure you are right?”—it’s “yes, I double-checked”. I bet GPT-3’s training data has huge numbers of examples of dialogue like this.

The chat thread we are currently engaging in will most likely be inside a model within the next year, too.

(Fortyseven is an alright dude.)

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#58

Post by mod50ack »

kevinpet wrote: post #38
londons_explore wrote: post #15

If I were the judge in this case, I would be looking to throw this lawyer in prison for a month, and ban him from ever being a lawyer again... Deliberately lying to the court, as a professional who should understand the consequences, in a way likely to not be detected, and likely to change the outcome of the case, ought to be met with a really strict punishment.

It's not clear he deliberately lied. He plausibly misunderstood a new tool he had recently been introduced to.

As a lawyer, though, he has a responsibility for the contents of his filings. And there are legal standards for how to research and verify these things — which he didn't follow.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#59

Post by Dalewyn »

grumple wrote: post #40
taberiand wrote: post #28

Earlier quoted context omitted.

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

"It doesn't lie, it just generates lies and printed them to the screen!" I don't think there's a difference.

An exceedingly complicated Autocomplete program, which an "AI" like ChatGPT is, does not have motives, does not know the concept of "lying" (nor any concept thereof), and simply does things as ordered by its user.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#60

Post by la64710 »

mehwoot wrote: post #7

No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? It did this because it's copying how humans talk, not what humans do. Humans say "I double checked" when asked to verify so…

ChatGPT did exactly what it is supposed to do. The lawyers who cited them are fools in my opinion. Of course OpenAI is also an irresponsible company to enable such a powerful technology without adequate warnings. With each chatGPT response they should provide citations (like Google does) and provide a clearly visible disclaimer that what it just spewed may be utter BS.

I only hope the judge passes an anecdotal order for all AI companies to include the above mentioned disclaimer with each of their responses.

Post reply on HN