Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lawyer cites fake cases invented by ChatGPT, judge is not amused

simonwillison.net

Post reply on HN
Print view View original

191–200 of 319 posts

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#191

Post by crooked-v »

hristov wrote: post #113

It is very concerning how people are anthropomorphizing chat gpt. It will get a lot of people into trouble. The media is largely to blame for this. Never mind most gushing media stories about AI, even the few stories criticizing AI treat it as a human. Chat GPT is racist, they say, it is sexist or a liar. Well it is neither of these things, because all of the above require consciousness and intent and it has none. It…

> information it has found somewhere

I think it's worth noting here that, without plugins active, ChatGPT doesn't 'find' anything - everything is just baked in as a single giant blob of vector data. That's why it has a specific date cutoff as to what it 'knows'.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#192

Post by dunham »

Wow, I asked Bard to summarize the NYT article and it completely changed the outcome:

> Sure. The article is about a man named Roberto Mata who sued Avianca Airlines after he was injured when a metal serving cart struck his knee during a flight to Kennedy International Airport in New York. His lawyer used a new language model called ChatGPT to help him with the case. ChatGPT is a large language model that can generate text, translate languages, write different kinds of creative content, and answer your questions in an informative way. It was trained on a massive dataset of text and code, and it can be used for a variety of tasks, including writing legal documents. In Mr. Mata's case, ChatGPT helped him to draft a lawsuit that was more persuasive and comprehensive than what he could have written on his own. The lawsuit was eventually successful, and Mr. Mata was awarded damages. This case shows the potential of ChatGPT and other language models to help lawyers with their work.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#193

Post by revertmean »

coldtea wrote: post #81
taberiand wrote: post #28

Earlier quoted context omitted.

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

> ChatGPT did not lie; it cannot lie. If it lies like a duck, it is a lying duck.

If it quacks like a duck, it's more likely that's it one of these https://www.amazon.co.uk/Zyyini-Whistle-Carring-Suitable-Bar...

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#194

Post by jimsimmons »

coffeebeqn wrote: post #148
jimsimmons wrote: post #49

Earlier quoted context omitted.

They should RLHF this behaviour out. Asking people to be aware of limitations is in similar vein as asking them to read ToC

If the model could tell when it was wrong it would be GPT-6 or 7. I think the best 4 could do is maybe it can detect when things enter the realm of the factual or mathematical etc and use a external service for that part

You have no basis to make that claim.

My point was a lot more subtle: if someone asks things like “double check it”, “are you sure” you can provide a template “I’m just a LM” response.

I’m not expecting the model to know what it doesn’t know. I’m not sure some future GPT variant can either

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#195

Post by jimsimmons »

akiselev wrote: post #83
jimsimmons wrote: post #53

Earlier quoted context omitted.

In the loop there indeed was a allegedly trained human in this instance

That's not what I would call in the loop. He didn't check that the sources were real. By "in the loop" I mean actively validating statements of fact generated by ChatGPT

Well they had every opportunity to inspect its output. Where in the loop and what gets inspected is not an easy problem to solve.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#196

Post by nemo44x »

whitemary wrote: post #175
nemo44x wrote: post #66

Earlier quoted context omitted.

Correct. ChatGPT is a bullshitter , not a liar. A bullshitter isn’t concerned with facts or truth or anything. A liar is concerned with concealing the truth. Bullshitters are actually probably worse than liars because at least liars live in the same reality as honest people.

Liars (humans) are often unconcerned with facts or truth either. More often these days, they are only concerned with profit. Are unconfirmed lies in the pursuit of mere profit somehow not lies?

This book [1] covers it.

[1] https://en.m.wikipedia.org/wiki/On_Bullshit#:~:text=The%20li....

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#197

Post by throwaway675309 »

rsynnott wrote: post #29
simonw wrote: post #17

Earlier quoted context omitted.

> And every response from ChatGPT should be preceded by a warning that it cannot be trusted. It kind of is - the ChatGPT site has this as a permanent fixture in the footer: > ChatGPT may produce inaccurate information about people, places, or facts. That's arguably ineffective though - even lawyers evidently don't read the small print in the footer!

Monty Python nicely addressed this, over 50 years ago. > Mr. Hilton: Oh, we use only the finest juicy chunks of fresh Cornish ram's bladder, emptied, steamed, flavoured with sesame seeds, whipped into a fondue, and garnished with lark's vomit. > Inspector: LARK'S VOMIT?!?!? > Mr. Hilton: Correct. > Inspector: It doesn't say anything here about lark's vomit! > Mr. Hilton: Ah, it does, on the bottom of the box, after '…

Humor aside I disagree. They are basically three types of people, the one who learns by reading, if you learn by observation, the rest just have to pee on the electric fence for themselves.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#198

Post by unusualmonkey »

coldtea wrote: post #128
Karellen wrote: post #123

Earlier quoted context omitted.

Lying implies intent, and knowing what the truth is. Saying something you believe to be true, but is wrong, is generally not considered a lie but a mistake. A better description of what ChatGPT does is described well by one definition of bullshit: > bullshit is speech intended to persuade without regard for truth. The liar cares about the truth and attempts to hide it; the bullshitter doesn't care if what they say is…

> Lying implies intent, and knowing what the truth is. Saying something you believe to be true, but is wrong, is generally not considered a lie but a mistake. Those are the semantics of lying. But "X like a duck" is about ignoring semantics, and focusing not on intent or any other subtletly, but only on the outward results (whether something has the external trappings of a duck). So, if it produces things that look l…

> So, if it produces things that look like lies, then it is lying.

Lying depends upon context.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#199

Post by pseudo0 »

agnosticmantis wrote: post #107
taberiand wrote: post #28

Earlier quoted context omitted.

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

Whether a statement is true or false doesn’t depend on the mechanism generating the statement. We should hold these models (or more realistically, their creators) to the same standard as humans. What do we do with a human that generates plausible-sounding sentences without regard for their truth? Let’s hold the creators of these models accountable, and everything will be better.

It seems like the appropriate party has been held responsible here - the lawyer who submitted false statements without doing proper verification and due diligence. This is no different than if the lawyer googled "case law supporting XYZ", found some random blog, and copy-pasted their citations without realizing they were made up.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#200

Post by majormajor »

joshjdr wrote: post #180
agnosticmantis wrote: post #145

Earlier quoted context omitted.

I don’t think the marketing around photoshop and chatgpt are similar. And that matters. Just like with self-driving cars, as soon as we hold the companies accountable to their claims and marketing, they start bringing the hidden footnotes to the fore. Tesla’s FSD then suddenly becomes a level 2 ADAS as admitted by the company lawyers. ChatGPT becomes a fiction generator with some resemblance to reality. Then I think…

I actually agree more with this comment more than after my initial read. You suggest some valid concerns about innovation that regulation could address. I guess the part I’m unsure about is the assertion about the dissimilarity to Photoshop, or if the marketing is the issue at hand. (E.g. did Adobe do a more appropriate job marketing with respect to conveying that their software is designed for the editing, but not d…

I think ChatGPT and Photoshop are both "designed for" the creation of novel things.

In Photoshop, though, the intent is clearly up to the user. If you edit that photo, you know you're editing the photo.

That's fairly different than ChatGPT where you ask a question and this product has been trained to answer you in a highly-confident way that makes it sound like it actually knows more than it does.

Post reply on HN