There's nothing wrong with starting with ChatGPT but the lazy ambulance chaser couldn't be bothered with a quick lookup in Westlaw. That's where things start going wrong.
Lawyer cites fake cases invented by ChatGPT, judge is not amused
31–40 of 319 posts
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#32Earlier quoted context omitted.
Also: https://news.ycombinator.com/item?id=36092509
Thanks! Macro-expanded: ChatGPT-Authored Legal Filing “Replete with Citations to Non-Existent Cases" https://news.ycombinator.com/item?id=36092509 (67 comments, 1 day ago)
dang is doing that frequently - but what does "macro-expanded" mean for normal users? Do you have a bookmarklet, extension or something similar for this or is there any secret incantation that makes HN generate such an entry automatically?
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#33> He had, he told Judge Castel, even asked the program to verify that the cases were real. It had said yes. It turns out, asking an unreliable narrator if it's being reliable is not a sound strategy.
Remember that professor who fed student essays to ChatGPT and asked if it wrote them. Mostly ChatGPT would reply yes, and the professor proceeded to fail the students.
https://news.ycombinator.com/item?id=35963163
("Texas professor fails entire class from graduating- claiming they used ChatGTP [sic]", 277 comments)
https://news.ycombinator.com/item?id=35980121
("Texas professor failed half of class after ChatGPT claimed it wrote their papers", 22 comments)
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#34I read this as malicious/intentional initially. Do people really think they can 100% believe ChatGPT?
Especially for someone like a lawyer I would expect to them verify any information they get from ChatGPT.
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#35I have asked ChatGPT to cite sources when I have doubts. I am often times able to confirm these sources. Seems this lawyer just took ChatGPT at its word without validating the cases.
Or maybe validated one or two, and then assumed they must all be correct.
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#36See, this is the sort of thing that makes me suspicious about the current AI hype (well, that, plus it’s largely the same people who were going on about cryptocurrency changing the world a few years ago); this stuff just isn’t very _useful_. Like, if writing plausible lies is the 2020s AI boom’s killer app, well, that’s not great, really.
Current AI is very useful in cases where the truth isn't very important. It's amazing for things like fantasy stories, video games. It also works in cases where existing tools can validate the bullshit AI generates (i.e. Copilot).
In cases where facts and sources are important, AI cannot be trusted. You can use it as long as you validate every single word it outputs, but at that point I do wonder what the point of using AI was in the first place.
It's also good at taking other existing work and creating new work out of it; not just for smart autocomplete tools like GPTs, but also for things like Stable Diffusion. Again, AI is incapable of attribution of sources, so that comes with obvious downsides, but in cases where the creator of the model have the necessary rights so they don't _need_ attribution to sell work (i.e. stock photo companies), it can be quite useful for generating things like filler images.
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#37In the other discussion of this topic, a lot of people said the lawyer should be disbarred, but personally I think people should be able to trust the tools marketed by trillion dollar tech companies, and a lot of the blame should be placed on Microsoft/OpenAI for overhyping ChatGPT and understating how likely it is to mislead. And every response from ChatGPT should be preceded by a warning that it cannot be trusted.
> And every response from ChatGPT should be preceded by a warning that it cannot be trusted. It kind of is - the ChatGPT site has this as a permanent fixture in the footer: > ChatGPT may produce inaccurate information about people, places, or facts. That's arguably ineffective though - even lawyers evidently don't read the small print in the footer!
Especially lawyers.
Half the job of lawyers is making people add useless warnings to everything that then everybody ignore.
May contain sesame. Your mileage may vary. All the characters are fictional.
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#38If I were the judge in this case, I would be looking to throw this lawyer in prison for a month, and ban him from ever being a lawyer again... Deliberately lying to the court, as a professional who should understand the consequences, in a way likely to not be detected, and likely to change the outcome of the case, ought to be met with a really strict punishment.
It's not clear he deliberately lied. He plausibly misunderstood a new tool he had recently been introduced to.
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#39I went ahead and asked ChatGPT with the browsing plugin [1] because I was curious and it answered that it was a real case citing an article about the fake citations! After some prodding ("Are you sure?") it spat out something slightly saner citing this very article!
> The case "Varghese v. China Southern Airlines Co., Ltd., 925 F.3d 1339 (11th Cir. 2019)" was cited in court documents, but it appears that there might be some confusion or controversy surrounding this citation. It was mentioned in a list of cases for which a lawyer was ordered to provide copies, according to a court order on leagle.com [2] . However, a blog post on simonwillison.net suggests that the case might not be genuine and that it might have been generated by a language model such as ChatGPT. The post discusses a situation where a lawyer might have used generated case citations in court documents without fully understanding the tool they were using. The post also includes screenshots where the language model appears to confirm the existence of the case [3].
The output is hilariously bad and it's depressing a licensed attorney actually pulled this crap.
This is just more evidence that ChatGPT should not be used for anything serious without a trained human in the loop.
[1] https://chat.openai.com/share/a6e27cf2-b9a6-4740-be2e-fdddab...
[2] https://www.leagle.com/decision/infdco20230414825
[3] https://simonwillison.net/2023/May/27/lawyer-chatgpt/ (The TFA!)
Re: Lawyer cites fake cases invented by ChatGPT, judge is not amused
#40No, it did not “double-check”—that’s not something it can do! And stating that the cases “can be found on legal research databases” is a flat out lie. What’s harder is explaining why ChatGPT would lie in this way. What possible reason could LLM companies have for shipping a model that does this? It did this because it's copying how humans talk, not what humans do. Humans say "I double checked" when asked to verify so…
ChatGPT did not lie; it cannot lie. It was given a sequence of words and tasked with producing a subsequent sequence of words that satisfy with high probability the constraints of the model. It did that admirably. It's not its fault, or in my opinion OpenAI's fault, that the output is being misunderstood and misused by people who can't be bothered understanding it and project their own ideas of how it should function…
"It doesn't lie, it just generates lies and printed them to the screen!"
I don't think there's a difference.