Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lawyer cites fake cases invented by ChatGPT, judge is not amused

simonwillison.net

Post reply on HN
Print view View original

301–310 of 319 posts

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#301

Post by joshjdr »

theamk wrote: post #284
joshjdr wrote: post #215

Earlier quoted context omitted.

If we’re moving past the marketing questions/concerns, I’m not sure I agree. For me, for now, ChatGPT remains a tool/resource, like: Google, Wikipedia, Photoshop, Adaptive Cruise Control, and Tesla FSD, (e.g. for the record despite mentioning FSD, I don’t think anyone should ever take a nap while operating a vehicle with any currently available technology). Did I miss when OpenAI marketed ChatGPT as a truthful resour…

I don't think it "legal matters" or not is important. OpenAI is marketing ChatGPT as accurate tool, and yet a lot of times it is not accurate at all. It's like.. imagine Wikipedia clone which claims earth is flat cheese, or a Cruise Control which crashes your car every 100th use. Would you call this "just another tool"? Or would it be "dangerously broken thing that you should stay away from unless you really know wha…

Did I miss when OpenAI marketed ChatGPT as a truthful resource?

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#302

Post by Mizoguchi »

"Anyone who has worked designing products knows that users don’t read anything—warnings, footnotes, any form of microcopy will be studiously ignored"

Users don't usually read long legal statements such as terms of services.

That's not the case of ChatGPT interface, the note about its limitations is clearly visible and very short.

This is as dumb as saying a city is at fault if someone drives into a clearly marked one way only street and causes an accident because people don't read anything.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#303

Post by coldtea »

imgabe wrote: post #243
coldtea wrote: post #118

Earlier quoted context omitted.

That's irrelevant to whether it lies like a duck or not. The expression "if it X like a duck" means precisely that we should judge a thing to be a duck or not, based on it having the external appereance and outward activity of a duck, and ignoring any further subleties, intent, internal processes, qualia, and so on. In other words, "it lies like a duck" means: if it produces things that look like lies, it is lying, a…

> we should judge a thing to be a duck or not, based on it having the external appereance and outward activity of a duck, and ignoring any further subleties, intent, internal processes, qualia, and so on. and the point here is we should not ignore further subtleties, intent, internal process, qualia, etc because they are extremely relevant to the issue at hand. Treating GPT like a malevolent actor that tells intentio…

>and the point here is we should not ignore further subtleties, intent, internal process, qualia, etc because they are extremely relevant to the issue at hand.

But the point is those are only relevant when trying to understand GPTs internal motivations (or lack thereof).

If we care for the practical effects of what it's spits out (the function the same as if GPT has lied to us), then calling them "hallucinations" is as good as calling them "lying".

>We do care how it got to produce incorrect information.

Well, not when trying to access whether it's true or false, and whether we should just blindly trust it.

From that practical aspect, most people care about (than about whether it has "intentions"), we can ignore any of its internal mechanics.

Thus treating it like it "beware, as it tends to lie", will have the same utility for most laymen (and be a much easier shortcut) than any more subtle formulation.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#304

Post by tankenmate »

cyanydeez wrote: post #254
tankenmate wrote: post #204

Earlier quoted context omitted.

But since this is about the law; ChatGPT can't lie because there is no mens rea. And of course this is a common failing with the common person when it comes to the law, a reckless disregard there of (until it is too late of course). And recklessness is also about intent, it is a form of wantonness, i.e. selfishness. This is why an insane person cannot be found guilty, you can't be reckless if you are incapable of dis…

Anyone who repeats chatgpt knowing is fallibility is lying, or deceitful. Chatgpt is a device unlike Wikipedia,

It's not lying if ChatGPT is correct (which it often is), so repeating ChatGPT isn't lying (since ChatGPT isn't always wrong); instead the behaviour is negligent or in the case of a lawyer grossly negligent since a lawyer should know better to check if it is correct before repeating it.

As always mens rea is a very important part of criminal law. Also, just because you don't like what someone says / writes doesn't mean it is a crime (even if it is factually incorrect).

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#305

Post by tankenmate »

dendriti wrote: post #231
Karellen wrote: post #184

Earlier quoted context omitted.

1) sounds like intent is present there? 2) "the camera cannot lie" - cameras have no intent? I feel like I'm missing something from those definitions that you're trying to show me? I don't see how they support your implication that one can ignore intent when identifying a lie. (It would help if you cited the source you're using.)

If I use ChatGPT to "hallucinate" a source, and post it here, am I lying?

either a) you knew it was false before posting, then yes you are lying. Or b) you knew there was a high possibility that ChatGPT could make things up, in which case you aren't lying per se, but engaging in reckless behaviour. If your job relies on you posting to HN, or you know and accept that others rely on what you post to HN then you are probably engaging in gross recklessness (like the lawyer in the article).

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#306

Post by ineedasername »

croes wrote: post #230
numpad0 wrote: post #190

Earlier quoted context omitted.

0) It calculates on data YOU SUPPLY. If the data is incomplete or incorrect, it tries its best to fill in blanks with plausible, but fabricated, data. You MAY NOT ask it an open ended or non-hypothetical question that require grounding beyond included in the input. e.g. “given following sentence, respond with the best summarization:, ” is okay; “what is a sponge cake” is not.

But the made us believe it an artificial intelligence. An intelligence knows which blanks can filled and which shouldn't without further information.

By that measure of intelligence even most humans, at some times, fail. Our brains misremember constantly, filling in details where information is lacking. One classic example are things like accidents and disasters. Accounts between people conflict, memories presenting events in an order that does not match that another’s memories, our outright fabrications. Dig up research on saccades and how our visual system does this on a constant basis, and can often be fooled as a result.

If knowing which blanks to fill in is a necessary condition of intelligence then all of humanity fails to measure up.

My point here is that very little is simple and straightforward. The concepts we use defy easy definitions. Our application of those concepts to artificial systems will inevitably do the same as a result.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#307

Post by threatofrain »

nemo44x wrote: post #253
threatofrain wrote: post #247

Earlier quoted context omitted.

It's not obvious that a bullshitter is "probably worse" than liars. Just because a bullshitter didn't care to research whether some vitamin pill meets marketing claims doesn't mean they're mentally volatile or psychotic. It's a bit of a leap to go from bullshit to asking whether a person lives in the same reality as everyone else.

The idea is that bullshitters are a greater enemy to the truth than liars because liars at least know the truth. You have to know the truth to lie about. Bullshitters have no concern for the truth at all and bullshit may or may not be true. The bullshitter doesn’t care, so long as their goal is made.

> Bullshitters have no concern for the truth at all and bullshit may or may not be true. The bullshitter doesn’t care, so long as their goal is made.

How is this different from a liar?

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#308

Post by wlonkly »

throwaway675309 wrote: post #183
coldtea wrote: post #118

Earlier quoted context omitted.

That's irrelevant to whether it lies like a duck or not. The expression "if it X like a duck" means precisely that we should judge a thing to be a duck or not, based on it having the external appereance and outward activity of a duck, and ignoring any further subleties, intent, internal processes, qualia, and so on. In other words, "it lies like a duck" means: if it produces things that look like lies, it is lying, a…

Abductive reasoning aside, people are already anthropomorphizing GPT enough without bringing in a loaded word like "lying" which implies intent. Hallucinates is a far more accurate word.

What bothers me about "hallucinates" is the removal of agency. When a human is hallucinating, something is happening to them that is essentially out of their control and they are suffering the effects of it, unable to tell truth from fiction, a dysfunction that they will recover from.

But that's not really what happens with ChatGPT. The model doesn't know truth from fiction in the first place, but the whole point of a useful LLM is that there is some level of control and consistency around the output.

I've been using "bullshitting", because I think that's really what ChatGPT is demonstrating -- not a disconnection from reality, but not letting truth get in the way of a good story.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#309

Post by majormajor »

joshjdr wrote: post #215
majormajor wrote: post #200

Earlier quoted context omitted.

I think ChatGPT and Photoshop are both "designed for" the creation of novel things. In Photoshop, though, the intent is clearly up to the user. If you edit that photo, you know you're editing the photo. That's fairly different than ChatGPT where you ask a question and this product has been trained to answer you in a highly-confident way that makes it sound like it actually knows more than it does.

If we’re moving past the marketing questions/concerns, I’m not sure I agree. For me, for now, ChatGPT remains a tool/resource, like: Google, Wikipedia, Photoshop, Adaptive Cruise Control, and Tesla FSD, (e.g. for the record despite mentioning FSD, I don’t think anyone should ever take a nap while operating a vehicle with any currently available technology). Did I miss when OpenAI marketed ChatGPT as a truthful resour…

> Did I miss when OpenAI marketed ChatGPT as a truthful resource for legal matters?

It's in the product itself. On the one hand, OpenAI says: "While we have safeguards in place, the system may occasionally generate incorrect or misleading information and produce offensive or biased content. It is not intended to give advice."

But at the same time, once you click through, the user interface is presented as a sort of "ask me anything" and they've intentionally crafted their product to take an authoritative voice regardless of if it's creating "incorrect or misleading" information. If you look at the documents submitted by the lawyer using it in this case, it was VERY confident about it's BS.

So a lay user who sees "oh occasionally it's wrong, but here it's giving me a LOT of details, this must be a real case" is understandable. Responsible for not double-checking, yes. I don't want to remove any blame from the lawyer.

Rather, I just want to also put some scrutiny on OpenAI for the impression created by the combination of their product positioning and product voice. I think it's misleading and I don't think it's too much to expect them to be aware of the high potential for mis-use that results.

Adobe presents Photoshop very differently: it's clearly a creative tool for editing and something like "context aware fill" or "generative fill" is positioned as "create some stuff to fill in" even when using it.

Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused

#310

Post by manquer »

NolF wrote: post #240
manquer wrote: post #129

Everyone is talking about ChatGPT , but is it not possible to train a model with only actual court documents and keep “temp” low and get accuracy levels as high or better than humans? Most legal (all formal really) documents are very predictably structured and should be easy to generate

Yes and no. Court decisions do generally follow a structure, but the decisions and the reasons for a final determination, may not always be clear. Judges also may throw in hypotheticals which whilst informative are not determinative. Once it gets to appellate courts and with different judges which may agree with the same outcome but on different grounds, it can get really hard to distinguish what is the test and the…

Not decisions by judges[1]!, filings by lawyers can be pretty proforma, in a ton of places they literally just fill your details and "generate" a filing.

Not every filing is generate-able for sure, however there are already tools which do create standard filings for human review, this would be just an enchantment covering some more use cases.

[1] It is vast gulf between generating a filing and generating a judgement. LLMs are not decision engines, generating text basis what is most likely from past data is one of worst ways we can be making decisions.

Post reply on HN