I have asked ChatGPT to cite sources when I have doubts.
I am often times able to confirm these sources.
Seems this lawyer just took ChatGPT at its word without validating the cases.
Lawyer cites fake cases invented by ChatGPT, judge is not amused
21–30 of 319 posts
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#22> No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? At what point does OpenAI (or any other company) become legally responsible for this kind of behavior from their LLM's? I…
Never?
Unless they advertise it as having the capability, it's got nothing to do with them.
If I hit someone with a hammer, that shit's on me, not the manufacturer.
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#23this is the future of lawyers, doctors, engineers, school teachers...
dark times are ahead.
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#24There’s a joke about “an LLB having an LLM” in here somewhere.
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#25In the other discussion of this topic, a lot of people said the lawyer should be disbarred, but personally I think people should be able to trust the tools marketed by trillion dollar tech companies, and a lot of the blame should be placed on Microsoft/OpenAI for overhyping ChatGPT and understating how likely it is to mislead. And every response from ChatGPT should be preceded by a warning that it cannot be trusted.
> And every response from ChatGPT should be preceded by a warning that it cannot be trusted. It kind of is - the ChatGPT site has this as a permanent fixture in the footer: > ChatGPT may produce inaccurate information about people, places, or facts. That's arguably ineffective though - even lawyers evidently don't read the small print in the footer!
Worse, it's buried in the middle of other fine print:
> Free Research Preview. ChatGPT may produce inaccurate information about people, places, or facts. ChatGPT May 24 Version
And it really understates the problem. It should say: Warning! ChatGPT is very likely to make shit up.
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#26See, this is the sort of thing that makes me suspicious about the current AI hype (well, that, plus it’s largely the same people who were going on about cryptocurrency changing the world a few years ago); this stuff just isn’t very _useful_. Like, if writing plausible lies is the 2020s AI boom’s killer app, well, that’s not great, really.
I disagree.
I've been tracking the many, many flaws in AI pretty closely (I wrote this article, and a bunch more in this series: https://simonwillison.net/series/llm-misconceptions/)
And yet... I'm finding ChatGPT and the like wildly useful on a personal level.
I think they're deceptively hard to use: you have to put in effort to learn them, and to learn how to avoid the many traps they set for you.
But once you've done that you can get very real productivity boosts from them. I use ChatGPT a dozen or so times a day, and I would be very sad to not have access to it any more.
I wrote a bit more about that here: https://simonwillison.net/2023/Mar/27/ai-enhanced-developmen... - and if anything this effect has got even stronger for me over the two months since I wrote that.
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#27> No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? At what point does OpenAI (or any other company) become legally responsible for this kind of behavior from their LLM's? I…
> At what point does OpenAI (or any other company) become legally responsible for this kind of behavior from their LLM's?
When they sell their bots to areas where lying is illegal. I.e., when a company pretends to do law.
OpenAI doesn't pretend ChatGPT is a lawyer and for good reason. The lawyer who decided to outsource his work is an idiot and can't shift blame to the tool he decided to abuse.
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#28No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? It did this because it's copying how humans talk, not what humans do. Humans say "I double checked" when asked to verify so…
ChatGPT did not lie; it cannot lie.
It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model.
It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function onto it.
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#29In the other discussion of this topic, a lot of people said the lawyer should be disbarred, but personally I think people should be able to trust the tools marketed by trillion dollar tech companies, and a lot of the blame should be placed on Microsoft/OpenAI for overhyping ChatGPT and understating how likely it is to mislead. And every response from ChatGPT should be preceded by a warning that it cannot be trusted.
> And every response from ChatGPT should be preceded by a warning that it cannot be trusted. It kind of is - the ChatGPT site has this as a permanent fixture in the footer: > ChatGPT may produce inaccurate information about people, places, or facts. That's arguably ineffective though - even lawyers evidently don't read the small print in the footer!
Monty Python nicely addressed this, over 50 years ago.
> Mr. Hilton: Oh, we use only the finest juicy chunks of fresh Cornish ram's bladder, emptied, steamed, flavoured with sesame seeds, whipped into a fondue, and garnished with lark's vomit.
> Inspector: LARK'S VOMIT?!?!?
> Mr. Hilton: Correct.
> Inspector: It doesn't say anything here about lark's vomit!
> Mr. Hilton: Ah, it does, on the bottom of the box, after 'monosodium glutamate'.
> Inspector: I hardly think that's good enough! I think it's be more appropriate if the box bore a great red label: 'WARNING: LARK'S VOMIT!!!'
> Mr. Hilton: Our sales would plummet!
Really, it should open every conversation with “by the way, I am a compulsive liar, and nothing I say can be trusted”. That _might_ get through to _some_ users.
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#30See, this is the sort of thing that makes me suspicious about the current AI hype (well, that, plus it’s largely the same people who were going on about cryptocurrency changing the world a few years ago); this stuff just isn’t very _useful_. Like, if writing plausible lies is the 2020s AI boom’s killer app, well, that’s not great, really.
You will see :)