Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lawyer cites fake cases invented by ChatGPT, judge is not amused

simonwillison.net

Post reply on HN
Print view View original

291–300 of 319 posts

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#291

Post by andrewfong »

taberiand wrote: post #28
mehwoot wrote: post #7

No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? It did this because it's copying how humans talk, not what humans do. Humans say "I double checked" when asked to verify so…

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

Well, it sort of is OpenAI's fault that it presented the interface as a chat bot though.

> It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model.

This is just autocorrect / autocomplete. And people are pretty good at understanding the limitations of generative text in that context (enough that "damn you autocorrect" is a thing). But for whatever reason, people assign more trust to conversational interfaces.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#292

Post by sickcodebruh »

dmix wrote: post #150
sickcodebruh wrote: post #105

Earlier quoted context omitted.

This is completely true but completely in conflict with how many very large companies advertise it. I’m a paid GitHub Copilot user and recently started using their chat tool. It lies constantly and convincingly, so often that I’m starting to wonder if it wastes more time than it saves. It’s simply not capable of reliably doing its job. This is on a “Tesla autopilot” level of misrepresenting a product but on a larger…

Where does Github misrepresent their Chat beta? On their marketing website?

Right from https://github.com/features/preview/copilot-x:

“Context aware conversations with your copilot. If you're stuck solving a problem, ask GitHub Copilot to explain a piece of code. Bump into an error? Have GitHub Copilot fix it. It’ll even generate unit tests so you can get back to building what’s next.”

This is almost a Homer Simpson running for garbage commissioner level of over-promising. I think Copilot is an incredible tool, what’s possible right now is amazing and it can save time and offer value. But the degree to which it doesn’t just fail but completely misdirects is at serious odds with the breathless marketing.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#293

Post by SZJX »

throwawayForMe2 wrote: post #185
masklinn wrote: post #46

Earlier quoted context omitted.

Funnily enough chatgpt had no more idea about that than about these legal cases, it lives in a state of perpetual hallucination and making stuff up is its only mode of operation.

It hallucinates a sequence of tokens, and we hallucinate meaning.

We with computer science knowledge understand this, but not the mass who are buying into the hype.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#295

Post by curiousgal »

kevinpet wrote: post #38
londons_explore wrote: post #15

If I were the judge in this case, I would be looking to throw this lawyer in prison for a month, and ban him from ever being a lawyer again... Deliberately lying to the court, as a professional who should understand the consequences, in a way likely to not be detected, and likely to change the outcome of the case, ought to be met with a really strict punishment.

It's not clear he deliberately lied. He plausibly misunderstood a new tool he had recently been introduced to.

"Your honour, I am not a cat"

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#296

Post by koonsolo »

dunham wrote: post #192

Wow, I asked Bard to summarize the NYT article and it completely changed the outcome: > Sure. The article is about a man named Roberto Mata who sued Avianca Airlines after he was injured when a metal serving cart struck his knee during a flight to Kennedy International Airport in New York. His lawyer used a new language model called ChatGPT to help him with the case. ChatGPT is a large language model that can generat…

So one language model lying to a human an sticking up for another (competing) language model.

It's starting people! ;)

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#297

Post by taberiand »

SantalBlush wrote: post #76
taberiand wrote: post #28

Earlier quoted context omitted.

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

It is designed to give the illusion that it reasons the way a human does, which is why many people are using it. To blame the average user--who quite obviously doesn't understand how LLMs work--isn't fair, either. A lawyer, however, should have vetted a new piece of tech before using it in this way.

Personally I think there should be restricted access to these models to people who have passed a basic course on their use and behaviour.

(I know that will never ever happen)

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#298

Post by taberiand »

theamk wrote: post #280
clnq wrote: post #142

Earlier quoted context omitted.

This harks back to around 1999 when people would often blame computers for mistakes in their math, documents, reports, sworn filings, and so on. Then, a thousand different permutations of "computers don't make mistakes" or "computers are never wrong" became popular sayings. Large Language Models (LLMs) are never wrong, and they do not make mistakes. They are not fact machines. Their purpose is to abstract knowledge a…

I just went to ChatGPT page, and was presented with the text: "ChatGPT: get instant answers, find creative inspiration, and learn something new. Use ChatGPT for free today." If something claims to give you answers, and those answers are incorrect, that something is wrong. Does not matter what it is -- model, human, dictionary, book. Claiming that their purpose is "to produce plausible language" is just wrong.. no one…

Producing plausible language is exactly what I use it for - mostly plausible blocks of code, and tedious work like rephrasing emails, generating docs, etc.

Everything it creates needs to be reviewed, particularly information that is outside my area of expertise. It turns out ChatGPT 4 passes those reviews extremely well - obviously too well given how many people are expecting so much more from it.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#299

Post by AmIDev »

What I have realized after using Bard(Palm2), ChatGPT(3.5) and some other LLMs is that they are good for tasks where an accuracy For example, labeling a million text samples with 90% accuracy by using few shot learning is a good use case. Writing a poem is good use case. Trying to learn a new language is not. Generating a small function that you can verify might be ok. Writing entire codebase is not.

So far, I haven't found any use case for personal use of LLMs. For work however, LLMs are going to be very useful with text(and potentially image) based machine learning tasks. Any tasks where having knowledge beyond the labeled training dataset is useful is going to be a good task for LLMs. One example is detecting fraud SMS.

Post reply on HN