Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lawyer cites fake cases invented by ChatGPT, judge is not amused

simonwillison.net

Post reply on HN
Print view View original

221–230 of 319 posts

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#221

Post by nemo44x »

whitemary wrote: post #203
nemo44x wrote: post #196

Earlier quoted context omitted.

This book [1] covers it. [1] https://en.m.wikipedia.org/wiki/On_Bullshit#:~:text=The%20li... .

No, it does not. I have read the full text and it was garbage to my mind; another nonsense apology for liberal society's degeneracy.

I’m curious how it apologizes for liberal societies degeneracy? I’ve read it as well and appreciate the nuance in differentiating the concepts regardless of which type of system you are in. For instance, the concepts are relevant in a reactionary system just as much as a degenerate progressive one.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#222

Post by taberiand »

RhysU wrote: post #207
clnq wrote: post #142

Earlier quoted context omitted.

This harks back to around 1999 when people would often blame computers for mistakes in their math, documents, reports, sworn filings, and so on. Then, a thousand different permutations of "computers don't make mistakes" or "computers are never wrong" became popular sayings. Large Language Models (LLMs) are never wrong, and they do not make mistakes. They are not fact machines. Their purpose is to abstract knowledge a…

> Large Language Models (LLMs) are never wrong, and they do not make mistakes. I call B.S. If LLMs never made mistakes we wouldn't train them. Any random initialization would work.

That's a different error context I think. It's a mistake if the model produces nonsense, because it's designed to produce realistic text. It's not a mistake if it produces non-factual information that looks realistic.

And it fundamentally cannot always produce factual information, it doesn't have that capacity (but then, neither do humans and with the ability to source information this statement may well be obsolete soon enough)

Though I wouldn't go so far as to say that the model cannot make mistakes - it clearly is susceptible to producing nonsense. I just think expecting it to always produce factual information is like using a hammer to cut wood and complaining the wood comes out all jagged

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#223

Post by matteoraso »

bobsmooth wrote: post #126
hristov wrote: post #113

It is very concerning how people are anthropomorphizing chat gpt. It will get a lot of people into trouble. The media is largely to blame for this. Never mind most gushing media stories about AI, even the few stories criticizing AI treat it as a human. Chat GPT is racist, they say, it is sexist or a liar. Well it is neither of these things, because all of the above require consciousness and intent and it has none. It…

It's really disconcerting how many people don't understand that these LLMs are programs not people.

It's because technological progress has outpaced our ability to process it, so we're like medieval peasants discovering a "Hello World" program and assuming the computer is literally greeting the world. Hopefully, people will learn to view LLMs as they really are before doing what the lawyer did en masse.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#224

Post by croes »

taberiand wrote: post #28
mehwoot wrote: post #7

No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? It did this because it's copying how humans talk, not what humans do. Humans say "I double checked" when asked to verify so…

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

>ChatGPT did not lie; it cannot lie.

More important it can't tell the truth either.

It produces the mostly likely series of words for the given prompt.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#225

Post by zztop44 »

daveguy wrote: post #217
taberiand wrote: post #28

Earlier quoted context omitted.

ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…

Corollary: ChatGPT did not tell the truth; it cannot tell the truth. If you accept the premise of the parent post, then this is a natural corollary. I accept the premise of the parent post.

Or more precisely: the truth or falsity of ChatGPT’s statements is incidental to the process by which it determines which statements to make.

Because unless you are using some personal definition of “tell the truth” then you must accept that ChatGPT often outputs statements which are demonstrably true.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#226

Post by croes »

disillusioned wrote: post #4

> He had, he told Judge Castel, even asked the program to verify that the cases were real. It had said yes. It turns out, asking an unreliable narrator if it's being reliable is not a sound strategy.

It is quite ironic that a lawyer made this mistake.

That's like asking the accused if he did it.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#227

Post by croes »

einpoklum wrote: post #146
jrockway wrote: post #42

Earlier quoted context omitted.

My main takeaway is that failing the second half of the class and misspelling ChatGPT leads to > 10x engagement.

My main takeway is that the guy who registers chatgtp.com is going to make a lot of money by providing bogus answers to frivolous questions :-)

There will be lots of data stealing ChatGPT plugins

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#228

Post by croes »

londons_explore wrote: post #11
disillusioned wrote: post #4

> He had, he told Judge Castel, even asked the program to verify that the cases were real. It had said yes. It turns out, asking an unreliable narrator if it's being reliable is not a sound strategy.

I wonder if this is a tactic so the court to deems this lawyer incompetent rather than giving the (presumably much harsher) penalty for deliberately lying to the court?

Maybe be a long run tactic to prevent future clients from switching to ChatGPT based solutions.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#229

Post by croes »

YetAnotherNick wrote: post #156
sickcodebruh wrote: post #105

Earlier quoted context omitted.

This is completely true but completely in conflict with how many very large companies advertise it. I’m a paid GitHub Copilot user and recently started using their chat tool. It lies constantly and convincingly, so often that I’m starting to wonder if it wastes more time than it saves. It’s simply not capable of reliably doing its job. This is on a “Tesla autopilot” level of misrepresenting a product but on a larger…

One of the only text written in chatgpt's own website that comes each time when you chat with it is "ChatGPT may produce inaccurate information about people, places, or facts."

That's a lot of words for just saying ChatGPT's answers can't be trusted.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#230

Post by croes »

numpad0 wrote: post #190
ineedasername wrote: post #79

Hilarious. It’s important to remember: 1) ChatGPT is not a research tool 2) It sort of resembles one and will absolutely act like one if you ask it to, and it it may even produce useful results! But… 3) You have to independently verify any factual statement it makes and also 4) In my experience the longer the chat session, the more likely it is to hallucinate, reiterate, and double down on previous output

0) It calculates on data YOU SUPPLY. If the data is incomplete or incorrect, it tries its best to fill in blanks with plausible, but fabricated, data. You MAY NOT ask it an open ended or non-hypothetical question that require grounding beyond included in the input. e.g. “given following sentence, respond with the best summarization:, ” is okay; “what is a sponge cake” is not.

But the made us believe it an artificial intelligence.

An intelligence knows which blanks can filled and which shouldn't without further information.

Post reply on HN