Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lawyer cites fake cases invented by ChatGPT, judge is not amused

simonwillison.net

Post reply on HN
Print view View original

41–50 of 319 posts

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#41

Post by leereeves »

londons_explore wrote: post #11
disillusioned wrote: post #4

> He had, he told Judge Castel, even asked the program to verify that the cases were real. It had said yes. It turns out, asking an unreliable narrator if it's being reliable is not a sound strategy.

I wonder if this is a tactic so the court to deems this lawyer incompetent rather than giving the (presumably much harsher) penalty for deliberately lying to the court?

Why assume malice? Asking ChatGPT to verify is exactly what someone who trusts ChatGPT might do.

I'm not surprised this lawyer trusted ChatGPT too much. People trust their lives to self driving cars, trust their businesses to AI risk models, trust criminal prosecution to facial recognition. People outside the AI field seem to be either far too trusting or far too suspicious of AI.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#42

Post by jrockway »

perihelions wrote: post #33
sampo wrote: post #13

Earlier quoted context omitted.

Remember that professor who fed student essays to ChatGPT and asked if it wrote them. Mostly ChatGPT would reply yes, and the professor proceeded to fail the students.

https://news.ycombinator.com/item?id=35963163 ( "Texas professor fails entire class from graduating- claiming they used ChatGTP [sic]", 277 comments) https://news.ycombinator.com/item?id=35980121 ( "Texas professor failed half of class after ChatGPT claimed it wrote their papers ", 22 comments)

My main takeaway is that failing the second half of the class and misspelling ChatGPT leads to > 10x engagement.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#43

Post by tmaly »

londons_explore wrote: post #35
tmaly wrote: post #21

I have asked ChatGPT to cite sources when I have doubts. I am often times able to confirm these sources. Seems this lawyer just took ChatGPT at its word without validating the cases.

Or maybe validated one or two, and then assumed they must all be correct.

I often want good data, so I validate everything.

ChatGPT tends to only give a limited number of results in the response.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#44

Post by jeroenhd »

londons_explore wrote: post #11
disillusioned wrote: post #4

> He had, he told Judge Castel, even asked the program to verify that the cases were real. It had said yes. It turns out, asking an unreliable narrator if it's being reliable is not a sound strategy.

I wonder if this is a tactic so the court to deems this lawyer incompetent rather than giving the (presumably much harsher) penalty for deliberately lying to the court?

I don't think the insanity plea works out well for lawyers. I'm not sure if "I'm too stupid to be a lawyer" is that much better than "I lied to the courts".

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#45

Post by perihelions »

jrockway wrote: post #42
perihelions wrote: post #33

Earlier quoted context omitted.

https://news.ycombinator.com/item?id=35963163 ( "Texas professor fails entire class from graduating- claiming they used ChatGTP [sic]", 277 comments) https://news.ycombinator.com/item?id=35980121 ( "Texas professor failed half of class after ChatGPT claimed it wrote their papers ", 22 comments)

My main takeaway is that failing the second half of the class and misspelling ChatGPT leads to > 10x engagement.

Err, out of abundance of caution, the misspelling of "ChatGPT" which I [sic]'d is original to the Texas A&M professor, who repeated the misspelling multiple times in his email/rant. The HN poster quoted the professor literally, and I am thus transitively [sic]'ing the professor – not the HN poster. I am not mocking an HN poster's typo.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#46

Post by masklinn »

sampo wrote: post #13
disillusioned wrote: post #4

> He had, he told Judge Castel, even asked the program to verify that the cases were real. It had said yes. It turns out, asking an unreliable narrator if it's being reliable is not a sound strategy.

Remember that professor who fed student essays to ChatGPT and asked if it wrote them. Mostly ChatGPT would reply yes, and the professor proceeded to fail the students.

Funnily enough chatgpt had no more idea about that than about these legal cases, it lives in a state of perpetual hallucination and making stuff up is its only mode of operation.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#47

Post by Buttons840 »

mehwoot wrote: post #7

No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? It did this because it's copying how humans talk, not what humans do. Humans say "I double checked" when asked to verify so…

GPT4 can double-check to an extent. I gave it a sequence of 67 letter As and asked it to count them. It said "100", I said "recount": 98, recount, 69, recount, 67, recount, 67, recount, 67, recount, 67. It converged to the correct count and stayed there.

This is quite a different scenario though, tangential to your [correct] point.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#48

Post by Bytewave81 »

grumple wrote: post #40
taberiand wrote: post #28

Earlier quoted context omitted.

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

"It doesn't lie, it just generates lies and printed them to the screen!" I don't think there's a difference.

To perhaps stir the "what do words really mean" argument, "lying" would generally imply some sort of conscious intent to bend or break the truth. A language model is not consciously making decisions about what to say, it is statistically choosing words which probabilistically sound "good" together.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#49

Post by jimsimmons »

simonw wrote: post #14
mehwoot wrote: post #7

No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? It did this because it's copying how humans talk, not what humans do. Humans say "I double checked" when asked to verify so…

Yeah, that was my conclusion too: What’s a common response to the question “are you sure you are right?”—it’s “yes, I double-checked”. I bet GPT-3’s training data has huge numbers of examples of dialogue like this.

They should RLHF this behaviour out.

Asking people to be aware of limitations is in similar vein as asking them to read ToC

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#50

Post by raincole »

grumple wrote: post #40
taberiand wrote: post #28

Earlier quoted context omitted.

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

"It doesn't lie, it just generates lies and printed them to the screen!" I don't think there's a difference.

Saying ChatGPT lies is like saying The Onion lies.
Post reply on HN