Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lawyer cites fake cases invented by ChatGPT, judge is not amused

simonwillison.net

Post reply on HN
Print view View original

311–319 of 319 posts

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#311

Post by flangola7 »

shagie wrote: post #201
flangola7 wrote: post #119

Earlier quoted context omitted.

ChatGPT isn't a legal entity but OpenAI is, and Altman has already recommend to Congress that coming regulations should make AI companies liable for produced text and be 230 exempt. I can see it already happening even without legislation, 230 shields liability from user-generated content but ChatGPT output isn't user generated. It's not even a recommendation algorithm steering you into other users' content telling wh…

Why should OpenAI be more liable for a tool that they've created than any other tool creator where the tool is intentionally misused and warnings on the tool ignored? If I wrote text in Microsoft Word and in doing so, I had a typo in (for example) the name of a drug that Word corrected to something that was incorrect, is Microsoft liable for the use of autocorrect? If I was copying and pasting data into excel and som…

ChatGPT is qualitatively different from any tool, like Microsoft Word. To suggest they are equivalent is so asinine as to not even warrant entertaining the idea.

A sophisticated word processor corrects your typos and grammar, a primitive language model by accident persuades you to kill yourself. Sam Altman, Christina Montgomery and Gary Marcus all testified to Congresy that Section 230 does not apply to their platforms. That will be extremely hard to defend when it eventually comes in front of a federal judge.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#312

Post by nemo44x »

threatofrain wrote: post #307
nemo44x wrote: post #253

Earlier quoted context omitted.

The idea is that bullshitters are a greater enemy to the truth than liars because liars at least know the truth. You have to know the truth to lie about. Bullshitters have no concern for the truth at all and bullshit may or may not be true. The bullshitter doesn’t care, so long as their goal is made.

> Bullshitters have no concern for the truth at all and bullshit may or may not be true. The bullshitter doesn’t care, so long as their goal is made. How is this different from a liar?

Liars know the truth and actively conceal it. Bullshitters don’t know or care what the truth is.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#313

Post by daveguy »

zztop44 wrote: post #225
daveguy wrote: post #217

Earlier quoted context omitted.

Corollary: ChatGPT did not tell the truth; it cannot tell the truth. If you accept the premise of the parent post, then this is a natural corollary. I accept the premise of the parent post.

Or more precisely: the truth or falsity of ChatGPT’s statements is incidental to the process by which it determines which statements to make. Because unless you are using some personal definition of “tell the truth” then you must accept that ChatGPT often outputs statements which are demonstrably true.

Yes, in this case "tell the truth" means knowingly tell the truth. Or knowingly lie. It just doesn't have a clue which is which.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#314

Post by threatofrain »

nemo44x wrote: post #312
threatofrain wrote: post #307

Earlier quoted context omitted.

> Bullshitters have no concern for the truth at all and bullshit may or may not be true. The bullshitter doesn’t care, so long as their goal is made. How is this different from a liar?

Liars know the truth and actively conceal it. Bullshitters don’t know or care what the truth is.

A bullshitter most certainly can care about the truth. A lot of people speak on the outer limits of their knowledge, such as when they talk about what's healthy or legal. That does not mean that they will carry on cynically after being informed.

The liar is cynical and is the one who sees the truth and tells you to go the wrong way.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#315

Post by golergka »

mehwoot wrote: post #7

No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? It did this because it's copying how humans talk, not what humans do. Humans say "I double checked" when asked to verify so…

> No, it did not “double-check”—that’s not something it can do!

It can with web plugin.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#316

Post by dragonwriter »

theamk wrote: post #283
throwaway675309 wrote: post #187

Earlier quoted context omitted.

No. What does this even mean? How would you make this actionable? LLM's are not "fact retrieval machines", and open AI is not presenting chat GPT as a legal case database. In fact they already have many disclaimers stating that GPT may provide information that is incorrect. If humans in their infinite stupidity choose to disregard these warnings, that's on them. Regulation is not the answer.

LLM's are not fact retrieval machines you say? but https://openai.com/product claims: "GPT-4 can follow complex instructions in natural language and solve difficult problems with accuracy." "use cases like long form content creation, extended conversations, and document search and analysis." and that's why we need regulations. In US, one needs FDA approval before claiming that a drug can treat some disease, the food…

GPT-4 can be used as an engine for document search and analysis, if you connect it to a database of documents to search and the right prompt to get it to search and analyze it.

The OpenAI chat frontend, for legal research, is not that.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#317

Post by pms »

taberiand wrote: post #28
mehwoot wrote: post #7

No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? It did this because it's copying how humans talk, not what humans do. Humans say "I double checked" when asked to verify so…

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

Their "own" ideas? Let me remind you that OpenAI released a report purposefully suggesting that GPT4 has relatively high IQ, passes a lot of college-level tests, and solves coding problems. Then it was revealed that there was training data contamination that led to good results in such tests [1], but GPT4 marketing received 10000 more attention than truth anyway. The popular belief is that using LLMs will give you a professional competitive advantage. Also, when we talk about the achievements of LLMs, then we anthropomorphize, but when we talk about their failures, then we don't anthropomorphize, i.e., "AI cannot lie"? Don't you see human biases drive AI hype?

In my opinion, people clearly are confused and misled by marketing and this isn't the first time it's happening. For instance, people were confused for 40+ about global warming, among others due to greenwashing campaigns [2]. Is it ok to mislead in ads? Are we supposed to purposefully take advantage of others by keeping them confused to gain a competitive advantage?

[1] https://twitter.com/cHHillee/status/1635790330854526981 [2] https://en.wikipedia.org/wiki/Global_Climate_Coalition

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#318

Post by taberiand »

pms wrote: post #317
taberiand wrote: post #28

Earlier quoted context omitted.

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

Their "own" ideas? Let me remind you that OpenAI released a report purposefully suggesting that GPT4 has relatively high IQ, passes a lot of college-level tests, and solves coding problems. Then it was revealed that there was training data contamination that led to good results in such tests [1], but GPT4 marketing received 10000 more attention than truth anyway. The popular belief is that using LLMs will give you a…

The context is people who should know better, whose job it is to put the effort into understanding the tools they are using.

Of course, I think these AI tools should require a basic educational course on their behaviour and operation before they can be used. But marketing nonsense is standard with everything; people have at least some responsibility for self education.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#319

Post by pms »

taberiand wrote: post #318
pms wrote: post #317

Earlier quoted context omitted.

Their "own" ideas? Let me remind you that OpenAI released a report purposefully suggesting that GPT4 has relatively high IQ, passes a lot of college-level tests, and solves coding problems. Then it was revealed that there was training data contamination that led to good results in such tests [1], but GPT4 marketing received 10000 more attention than truth anyway. The popular belief is that using LLMs will give you a…

The context is people who should know better, whose job it is to put the effort into understanding the tools they are using. Of course, I think these AI tools should require a basic educational course on their behaviour and operation before they can be used. But marketing nonsense is standard with everything; people have at least some responsibility for self education.

> Marketing nonsense is standard

Definitely, it will be standard, until we start pointing out lies.

Post reply on HN