Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lawyer cites fake cases invented by ChatGPT, judge is not amused

simonwillison.net

Post reply on HN
Print view View original

211–220 of 319 posts

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#211

Post by numpad0 »

ChatGPT just isn’t going to say the right thing, or thing grounded to the baseline reality, beyond what it cannot help but doing so restrained by your input.

The only connection between it and this world is your input. ChatGPT is floating in the heavens, and you’re grounding it by at most a fishing line, through the textbox. It has to be framed as such. People praising it as a next gen search engine[that finds a data from database] is(perhaps this is the word that best fit the situation!), hallucinating.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#212

Post by shagie »

icedchai wrote: post #141
MattGaiser wrote: post #110

Earlier quoted context omitted.

> What do we do with a human that generates plausible-sounding sentences without regard for their truth? Elect them as leaders?

ChatGPT is perfect for generating company mission statements, political rhetoric, and other forms of BS.

In Anathem there's the word: Bulshytt

    Bulshytt: Speech (typically but not necessarily commercial or political) that employs euphemism, convenient vagueness, numbing repetition, and other such rhetorical subterfuges to create the impression that something has been said.
(there's quite a bit more about it to be said, though quoting it out of context loses much of the world building associated with it... and the raw quote is riddled with strange spellings that would have even more confusion)

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#213

Post by simonw »

dunham wrote: post #192

Wow, I asked Bard to summarize the NYT article and it completely changed the outcome: > Sure. The article is about a man named Roberto Mata who sued Avianca Airlines after he was injured when a metal serving cart struck his knee during a flight to Kennedy International Airport in New York. His lawyer used a new language model called ChatGPT to help him with the case. ChatGPT is a large language model that can generat…

Oh dear!

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#214

Post by olalonde »

coldtea wrote: post #118
olalonde wrote: post #112

Earlier quoted context omitted.

It doesn't lie like a duck. It unintentionally says falsehoods. Lying is intentional.

That's irrelevant to whether it lies like a duck or not. The expression "if it X like a duck" means precisely that we should judge a thing to be a duck or not, based on it having the external appereance and outward activity of a duck, and ignoring any further subleties, intent, internal processes, qualia, and so on. In other words, "it lies like a duck" means: if it produces things that look like lies, it is lying, a…

I know what the expression means and tend to agree with the duck test. I just disagree that ChatGPT passes the "lying duck" test. A "lying duck" would be more systematic and consistent in its output of false information. ChatGPT occasionally outputs incorrect information, but there's no discernable motive or pattern, it just seems random and unintentional.

If it looked like ChatGPT was intentionally being deceptive, it would be a groundbreaking discovery, potentially even prompting a temporary shutdown of ChatGPT servers for a safety assessment.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#215

Post by joshjdr »

majormajor wrote: post #200
joshjdr wrote: post #180

Earlier quoted context omitted.

I actually agree more with this comment more than after my initial read. You suggest some valid concerns about innovation that regulation could address. I guess the part I’m unsure about is the assertion about the dissimilarity to Photoshop, or if the marketing is the issue at hand. (E.g. did Adobe do a more appropriate job marketing with respect to conveying that their software is designed for the editing, but not d…

I think ChatGPT and Photoshop are both "designed for" the creation of novel things. In Photoshop, though, the intent is clearly up to the user. If you edit that photo, you know you're editing the photo. That's fairly different than ChatGPT where you ask a question and this product has been trained to answer you in a highly-confident way that makes it sound like it actually knows more than it does.

If we’re moving past the marketing questions/concerns, I’m not sure I agree.

For me, for now, ChatGPT remains a tool/resource, like: Google, Wikipedia, Photoshop, Adaptive Cruise Control, and Tesla FSD, (e.g. for the record despite mentioning FSD, I don’t think anyone should ever take a nap while operating a vehicle with any currently available technology).

Did I miss when OpenAI marketed ChatGPT as a truthful resource for legal matters?

Or is this not just an appropriate story that deserves retelling to warn potential users about how not to misappropriate this technology?

At the end of the day, for an attorney, a legal officer of the court, to have done this is absolutely not the technology’s, nor marketing’s, fault.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#216

Post by daveguy »

taberiand wrote: post #28
mehwoot wrote: post #7

No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? It did this because it's copying how humans talk, not what humans do. Humans say "I double checked" when asked to verify so…

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

Corollary: ChatGPT did not tell the truth; it cannot tell the truth.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#217

Post by daveguy »

taberiand wrote: post #28
mehwoot wrote: post #7

No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? It did this because it's copying how humans talk, not what humans do. Humans say "I double checked" when asked to verify so…

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

Corollary: ChatGPT did not tell the truth; it cannot tell the truth.

If you accept the premise of the parent post, then this is a natural corollary.

I accept the premise of the parent post.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#218

Post by LASR »

I’ve said this many times: stop using ChatGPT as a database. It does not contain facts.

It may appear to contain some facts. Some may also be actually true.

The truly useful usecase is as a reasoning engine. You can paste in a document and ask some questions about the facts in that document. Then it does a much better job, enough to be actually useful.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#219

Post by kristjansson »

grumple wrote: post #40
taberiand wrote: post #28

Earlier quoted context omitted.

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

"It doesn't lie, it just generates lies and printed them to the screen!" I don't think there's a difference.

Language generated without regard to its truth value is different than language that cares about its truth value. Ask Harry Frankfurt.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#220

Post by wang_li »

shagie wrote: post #201
flangola7 wrote: post #119

Earlier quoted context omitted.

ChatGPT isn't a legal entity but OpenAI is, and Altman has already recommend to Congress that coming regulations should make AI companies liable for produced text and be 230 exempt. I can see it already happening even without legislation, 230 shields liability from user-generated content but ChatGPT output isn't user generated. It's not even a recommendation algorithm steering you into other users' content telling wh…

Why should OpenAI be more liable for a tool that they've created than any other tool creator where the tool is intentionally misused and warnings on the tool ignored? If I wrote text in Microsoft Word and in doing so, I had a typo in (for example) the name of a drug that Word corrected to something that was incorrect, is Microsoft liable for the use of autocorrect? If I was copying and pasting data into excel and som…

>Why should OpenAI be more liable for a tool that they've created than any other tool creator where the tool is intentionally misused and warnings on the tool ignored?

Because they provide a service, not a tool.

Post reply on HN