Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

arstechnica.com

Post reply on HN
Print view View original

141–150 of 212 posts

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#141

Post by alfalfasprout »

Zetice wrote: post #130

So how do LLMs fit into the legal profession, if at all? Do legal tools that make use of LLMs just need to come with big ol' disclaimers at the top saying, "This tool does not represent a legal opinion, please verify the output independently."?

At minimum, attorneys need to review the work's citations and wording for accuracy.

At the end of the day, this is not too different than LLMs consistently writing subtly broken code-- someone needs to comb through it and fix it.

We're currently in the phase where the potential of LLMs is suddenly appealing to many but where most people don't quite understand it's not really magic and that even after they evolve they will remain critically flawed. Expect serious growing pains as a result.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#142

Post by roflyear »

okasaki wrote: post #95
gerdesj wrote: post #75

Earlier quoted context omitted.

> Or, learn how to say “I don’t know” It doesn't know that it doesn't know! It is, very roughly speaking, a model that is designed to print out the most likely word given its current input and training, and then the next word etc. Whereas you or I might be mistaken about some of our faculties, memories and skills, ChatGPT cannot possibly "know" what its limitations are. It was never taught what it was not taught (obv…

It seems that you don't know what you don't know, really. There's no way to definitively know what properties ChatGPT has. It does seem to reason to some extent and it does often say that some information isn't known/there's no data. And it almost obnoxiously often tells you that it's simplifying a complex and multifaceted situation.

It doesn't know things. I promise.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#143

Post by alfalfasprout »

thefourthchime wrote: post #12

Sigh, I'm getting very sick of hearing about how "ChatGPT" makes stuff up. Yes, 3.5 made a lot of stuff up, 4.0 still does, but it's much rarer. I wish people would mention this, it's all treated as the same thing. It's like talking about how unreliable these "Airplanes" are when they are talking about prop planes, even though jets are out.

except ChatGPT 4 makes a ton of stuff up too. Rarer? Slightly. But the stuff it makes up is even more plausible sounding and harder to catch.

The improvements, in practice, are in the stuff it doesn't hallucinate. LLMs as a whole are still to be treated with great care.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#144

Post by clnq »

Zetice wrote: post #130

So how do LLMs fit into the legal profession, if at all? Do legal tools that make use of LLMs just need to come with big ol' disclaimers at the top saying, "This tool does not represent a legal opinion, please verify the output independently."?

The same way they fit into all professions demanding accountability and precision - they can be used for exploration but everything they say should be verified.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#145

Post by zerohp »

oblio wrote: post #129
jimnotgym wrote: post #30

Earlier quoted context omitted.

> Or, learn how to say “I don’t know” This is the correct answer. It is like a sad salesman who is out of his depth, but decides to keep bullshiting!

Well, that tells you a lot about: 1. The people designing it (either optimists or looking for a quick exit). 2. The learning set they're using, which I believe is some kind of internet crawl of sorts? I imagine humanity, as a whole, bullshits its way through most of its life.

It is almost certainly case 2. People don't often write text on the internet just to say that they don't know. It doesn't even make sense to post that most of the time.

A great example would be on a Q/A forum or something like Stackoverflow. It better to let someone else answer when you don't know.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#146

Post by gumballindie »

BugsJustFindMe wrote: post #106
cj wrote: post #19

I asked ChatGPT to tell me a riddle. It was “What is always hungry, needs to be fed, and makes your hands red?” (Or something like that) I asked for a hint about 5 times and it kept giving more legitimate sounding hints. Finally I gave up and asked for the answer to the riddle, and it spit out a random fruit which made no sense as the answer to the riddle. I then repeated the riddle and asked ChatGPT what the answer…

I need to know what version of ChatGPT you were using, because this is a critical piece of information that everyone just blatantly ignores, and I can only imagine that it's out of ignorance of the significance of the difference. This is what happened when I asked ChatGPT 4... ME Give me hints without outright telling me the answer to the riddle: "What is always hungry, needs to be fed, and makes your hands red?" Cha…

Here is what poe told me:

“Think about things that you might need to feed regularly, especially things that might leave stains on your hands. It might be something that you encounter frequently in your daily life.”

“ The answer to the riddle is "fire". Fire is always hungry for fuel, needs to be fed with wood or other combustible materials, and can make your hands red if you handle it or if you're cooking food over an open flame.”

And second time:

“This thing is a natural phenomenon that can occur in different situations and locations. It is often associated with warmth and light, and it can be both helpful and dangerous. People have been using it for various purposes for thousands of years.”

“The answer to the riddle is "fire". Fire is a natural phenomenon that needs fuel to keep burning, and it produces heat and light that can make your hands red. Fire has been used by humans for various purposes for thousands of years, such as cooking, heating, and providing light. However, fire can also be dangerous if not handled carefully.”

I am amazed at how a free bot is basically as good, if not better, as a openai’s bot. I am quite certain locally run llms will also be able to outcompete chatgpt. Probably why sam altman is desperate to block them.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#147

Post by subarctic »

hluska wrote: post #138
dylan604 wrote: post #111

Earlier quoted context omitted.

no, but that driver might just be that much more successful if they get away with it. in fact, while not a mountain hairpin as your example, there was a recent race car driver that did a similar thing by intentionally using the wall as a push back to allow a maneuver that allowed for success. so a clever comment attempting to prove a point is not always indicative of a proven point ;-)

No, but a winky smile after an intentionally inflammatory reply is usually indicative of someone I’m not interested in reading again. Doubly so when they refuse to use capital letters.

reading what again?

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#148

Post by delecti »

Zetice wrote: post #130

So how do LLMs fit into the legal profession, if at all? Do legal tools that make use of LLMs just need to come with big ol' disclaimers at the top saying, "This tool does not represent a legal opinion, please verify the output independently."?

I think LLMs should be used with basically the same stipulations in any field. The words it outputs are usually valid English, but not necessarily accurate, so it's good for brainstorming but needs to be fact-checked. Overall, whether it's useful depends on whether the time required for the latter is less than the time you save with the former. Personally, I've found them most useful as a way to provoke myself into Cunningham's Law, essentially relying on the fact that they make shit up. https://meta.wikimedia.org/wiki/Cunningham%27s_Law

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#149

Post by cj »

BugsJustFindMe wrote: post #106
cj wrote: post #19

I asked ChatGPT to tell me a riddle. It was “What is always hungry, needs to be fed, and makes your hands red?” (Or something like that) I asked for a hint about 5 times and it kept giving more legitimate sounding hints. Finally I gave up and asked for the answer to the riddle, and it spit out a random fruit which made no sense as the answer to the riddle. I then repeated the riddle and asked ChatGPT what the answer…

I need to know what version of ChatGPT you were using, because this is a critical piece of information that everyone just blatantly ignores, and I can only imagine that it's out of ignorance of the significance of the difference. This is what happened when I asked ChatGPT 4... ME Give me hints without outright telling me the answer to the riddle: "What is always hungry, needs to be fed, and makes your hands red?" Cha…

If I were still able to edit my original comment, I would add a note at the bottom that says to take the experience as a casual person downloading an AI app after hearing about it on the news.

Such as a lawyer who’s not particularly tech savvy.

The main point is it’s irresponsible to trust LLM output for any critical/important purpose because it’s not perfect. But too many first time users think it is perfect and trustworthy at face value, when it’s not.

I don’t actually know the version since I was interacting via an unofficial iOS app using some LLM under the hood. It may not have even been ChatGPT.

Re: Lawyer cited 6 fake cases made up by ChatGPT; judge calls it “unprecedented”

#150

Post by gerdesj »

okasaki wrote: post #95
gerdesj wrote: post #75

Earlier quoted context omitted.

> Or, learn how to say “I don’t know” It doesn't know that it doesn't know! It is, very roughly speaking, a model that is designed to print out the most likely word given its current input and training, and then the next word etc. Whereas you or I might be mistaken about some of our faculties, memories and skills, ChatGPT cannot possibly "know" what its limitations are. It was never taught what it was not taught (obv…

It seems that you don't know what you don't know, really. There's no way to definitively know what properties ChatGPT has. It does seem to reason to some extent and it does often say that some information isn't known/there's no data. And it almost obnoxiously often tells you that it's simplifying a complex and multifaceted situation.

Its a model that takes an input and spits out the most likely output given its training.

"There's no way to definitively know what properties ChatGPT has." - yes there is: ask it how the war in Ukraine is progressing or some other time based thing. It stops in 2021.

It is a really useful tool but it isn't sentient.

Post reply on HN