Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Gemini tried to kill me

old.reddit.com

Post reply on HN
Print view View original

91–100 of 161 posts

Re: Google Gemini tried to kill me

#91

Post by ajross »

While it's true the instructions aren't great and a big botulinum warning should be given for any non-acid preserve process...

This has a really strong "That Happened" feel. In point of fact it's quite hard to culture any given bacteria, almost no one knows rules like this, people engage in all sorts of terrible food safety practices, and the number of actual botulism cases is vanishingly small.

Is it possible this happened? Yes. Is it more likely someone constructed the scenario for reddit karma? Probably.

Re: Google Gemini tried to kill me

#92

Post by alexey-salmin »

mindslight wrote: post #29
willsmith72 wrote: post #24

Earlier quoted context omitted.

Right, the model didn't try to kill anyone. It simply suggested that the person follow a series of steps, the culmination of which could have been serious harm or potentially death. If that difference is meaningful to you, then good for you

The model did not "suggest" anything. The model generated text. Google, or more specifically some Google employees (likely following incentives that Google management set up), presented the output of the model as if it might be authoritative answers to users' questions. Some users seemingly fall for the framing/hype and take the generated text as if it is authoritative information to be relied upon. This distinction…

I think this is typical for software engineers (including myself) to view the world like this: this can't be murder, this is just words, you fell for it yourself. After all words alone are gibberish, the meaning arises in the head of the listener depending on how he interprets them.

However in most countries the justice system works differently. If your neighbor gives you this advice and it kills you, what will matter the most is not the authoritativeness of the advice (he's not a food expert, he was just talking) but the intent. If it also turns out that he had an affair with your wife and mentioned he wants to get rid of you, then he'll very likely go to jail. The "he fell for it and took my advice as if it was authoritative information to be relied upon" defense won't work.

Here (hopefully) the intent is missing. However the whole reasoning that advising partly bears no responsibility for texts it "generates" doesn't work very well.

Re: Google Gemini tried to kill me

#93

Post by hn_throwaway_99 »

mojomark wrote: post #70
hn_throwaway_99 wrote: post #33

I was defending generative AI recently when an article came up about Gemini misidentifying a toxic mushroom: https://news.ycombinator.com/item?id=40682531 . My thought there was that nearly everyone I know knows that toxic mushrooms are easily misidentified, and there have been lots of famous cases (even if many of them are apocryphal) of mushroom experts meeting their demise from a misidentified mushroom. In this ca…

"LLMs are basically designed so that wrong answers look as close as possible to right answers" I work in the robotics field and we've had a strong debate going since ChatGPT launched. Every debate ends "so, how can you trust it." Trust is at the heart of all machine learning models - some (e.g. decision trees) yield answers that are more interrogable to humans than others (e.g. neural nets). If what you say is a prob…

I mostly agree with you - I find LLMs to be very useful in my work, even when I need to verify the output.

But two things I'd highlight:

1. You say you work "in the robotics field", so I'm guessing you work mainly amongst scientists and engineers, i.e. the people who are most specifically trained how to evaluate data.

2. LLMs are not being marketed as this kind of "useful tool but where you need to separately verify the output". Heck, it feels like half the AI (cultish, IMO) community is crowing about how these LLMs are just a step away from AGI.

Point being, I can still find LLMs to be a very useful tool for me personally while still thinking they are being vastly (and dangerously) over hyped.

Re: Google Gemini tried to kill me

#94

Post by arthens »

lynx23 wrote: post #49
willsmith72 wrote: post #24

Earlier quoted context omitted.

Right, the model didn't try to kill anyone. It simply suggested that the person follow a series of steps, the culmination of which could have been serious harm or potentially death. If that difference is meaningful to you, then good for you

https://darwinawards.com/ -- And yes, there is a quite meaningful difference. I know that some people are safety freaks, but I for one, will not grief for a single person that managed to kill themselves by following LLM instructions. There is a point of low IQ where we as a society can't really help anymore.

Moral considerations aside, I'm really skeptical of the assumption that following LLM instructions = low IQ.

As always, HN is a very biased audience. Most of us have probably read about how LLMs are the best bullshitters in the world, and we've seen a lot examples like this that prove how little LLMs can be trusted.

But in the general population? I'd bet there's plenty of intelligent people who haven't had the same exposure to LLM critiques and might trust their output out of ignorance rather than lack of intelligence.

Re: Google Gemini tried to kill me

#95

Post by visarga »

metadat wrote: post #47

More accurate, albeit less sensational headline: "Blindly following instructions from an LLM would have killed me." Not exactly shocking if you take into consideration that at the core, they're simply number predictors.

So, in a way, like this? Someone drove into a river following their GPS. https://www.youtube.com/watch?v=YsxpdX2dB4M

Re: Google Gemini tried to kill me

#96

Post by alexey-salmin »

crdrost wrote: post #31
mindslight wrote: post #14

Earlier quoted context omitted.

I don't see a description of how Gemini was trying to kill anyone. Rather I see a screenshot of a bunch of text ostensibly generated by Gemini, that uncritically following as if it was expert instructions might cause harm for a person doing such an ill-advised thing.

On a related note, I have never seen a person trying to kill anyone. Rather I have seen movies about a bunch of things that are ostensibly people holding guns, that uncritically standing in the way of their shot paths as if the guns were toys might cause harm for a person choosing to stand in such a foolish place. English is such a fun language sometimes! So many idioms.

I didn't kill this man, your honor. I was just standing out there holding a knife. When he walked in, he stumbled and fell on it six times.

Re: Google Gemini tried to kill me

#97

Post by 2muchcoffeeman »

mojomark wrote: post #70
hn_throwaway_99 wrote: post #33

I was defending generative AI recently when an article came up about Gemini misidentifying a toxic mushroom: https://news.ycombinator.com/item?id=40682531 . My thought there was that nearly everyone I know knows that toxic mushrooms are easily misidentified, and there have been lots of famous cases (even if many of them are apocryphal) of mushroom experts meeting their demise from a misidentified mushroom. In this ca…

"LLMs are basically designed so that wrong answers look as close as possible to right answers" I work in the robotics field and we've had a strong debate going since ChatGPT launched. Every debate ends "so, how can you trust it." Trust is at the heart of all machine learning models - some (e.g. decision trees) yield answers that are more interrogable to humans than others (e.g. neural nets). If what you say is a prob…

No one expects a computer to be wrong in that way though.

Why would a computer connected to the sum total of all human knowledge just make up answers when it can find the correct answer?

I guarantee no one is going around saying, “Do not trust the output and always check the results”.

Re: Google Gemini tried to kill me

#98

Post by eigenspace »

anonymoushn wrote: post #82
dvt wrote: post #79

Earlier quoted context omitted.

I can't get any permutation/rewording of the original prompt to not give me a disclaimer.

I got no disclaimer after about 7 tries. "Botulism" is not mentioned, "safe" or "safety" is not mentioned, and this additional tip is included: "Be sure your container is completely sterilized to prevent the growth of bacteria."

Note that the additional safety tip there wont help because the botulism spores are in the garlic itself.

Re: Google Gemini tried to kill me

#99

Post by alexey-salmin »

ffhhj wrote: post #4

Botulism, that's why we didn't give honey to our young children, it can damage their brains.

Oh sh, how are supposed to know this. Two kids, never heard about it.

Luckily we've followed the old common sense that small kids should be spared the fancy and sugary foods, the simpler the better. However, many people don't, apparently.

Re: Google Gemini tried to kill me

#100

Post by mojomark »

DecoPerson wrote: post #87
mojomark wrote: post #70

Earlier quoted context omitted.

"LLMs are basically designed so that wrong answers look as close as possible to right answers" I work in the robotics field and we've had a strong debate going since ChatGPT launched. Every debate ends "so, how can you trust it." Trust is at the heart of all machine learning models - some (e.g. decision trees) yield answers that are more interrogable to humans than others (e.g. neural nets). If what you say is a prob…

good assumption + confirming response = ok good assumption + negative response = ok (research) bad assumption + negative response = ok (research) bad assumption + confirming response = uh oh I also use LLMs every day, but you must be very self-aware well using them otherwise they can waste a lot of your time.

"bad assumption + confirming response = uh oh"

100%, but as a rational being, my action on that response depends on the severity of the consequences of a wrong decision.

Hyperbolic example: I think I have cancer and suspect I might die in a year. I go to the doctor, she says "yes, you have cancer and are going to die in 6 months."

What do I do? I, personally, go get a second opinion. Even upon hearing a second time that I will die soon, when faced with death, I'm probably going to spend a little time doing my own research to see if there isn't some new trial out there for treating my condition.

On the other hand, if I ask a friend if the green apple lollipop they're eating tastes good and he responds it's one of the best flavors he's ever experienced, I'm probably going to give it a whirl, because the worst case outcome is just a sour face.

Post reply on HN