Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini AI tells the user to die

tomshardware.com

Post reply on HN
Print view View original

61–70 of 150 posts

Re: Gemini AI tells the user to die

#61

Post by Gigachad »

crishoj wrote: post #16
LeoPanthera wrote: post #10

I’m willing to be wrong, but, I don’t believe it. The user’s inputs are so weird, and the response is so out of left field… I would put money on this being faked somehow, or there’s some missing information. Edit: Yes even with the Gemini link, I’m still suspicious. It’s just too sci-fi.

The conversation is up on Gemini still in its entirety: https://gemini.google.com/share/6d141b742a13 Nothing out of the ordinary, except for that final response.

The whole conversation thread is weird. But it doesn’t look like they coerced the response. It’s just so random.

Re: Gemini AI tells the user to die

#62

Post by [deleted] »

surgical_fire wrote: post #51

Of all generative AI blunders, and it has plenty, this one is perhaps one of the least harmful ones. I mean, I can understand that someone might be distressed by reading it, but at the same time, once you understand it is just outputting text from training data, you can dismiss it as a bullshit response, probably tied to a bad prompt. Much worse than that, and what makes Generative AI very useless to me, is its prope…

[deleted]

Re: Gemini AI tells the user to die

#63

Post by Terr_ »

ryaniscool wrote: post #53
notepad0x90 wrote: post #43

I have yet to jump on the LLM train (did it leave without me?), but I disagree on this sort of " does/says ". Understand the technology and use it accordingly. it is not a person. If ChatGPT or Gemini output some incorrect statement, guess what? it is a hallucination, error or whatever you want to call it. treat it as such and move on. This pearl-clutching, I am concerned, will only result in the models being heavily…

Yeah, I find the shock and indignant outrage at a computer program's output to be disturbing. "AI safety" is clever marketing. It implies that these are powerful entities when really they are just upgraded search engines. They don't think, they don't reason. The token generator chose an odd sequence this time.

Ouija-board safety. Sometimes it hallu--er, it channels the wrong spirits from the afterlife. But don't worry, the rest of the time it is definitely connecting to the correct spirits from beyond the veil.

Re: Gemini AI tells the user to die

#64

Post by jfoster »

LeoPanthera wrote: post #10

I’m willing to be wrong, but, I don’t believe it. The user’s inputs are so weird, and the response is so out of left field… I would put money on this being faked somehow, or there’s some missing information. Edit: Yes even with the Gemini link, I’m still suspicious. It’s just too sci-fi.

If it were fake, I don't think Google would issue this statement to CBS News:

"Large language models can sometimes respond with non-sensical responses, and this is an example of that. This response violated our policies and we've taken action to prevent similar outputs from occurring."

https://www.cbsnews.com/news/google-ai-chatbot-threatening-m...

Re: Gemini AI tells the user to die

#65

Post by svantana »

surgical_fire wrote: post #60
svantana wrote: post #58

Earlier quoted context omitted.

The raw language models will always have strange edge cases, for sure. But chat services are systems, and they almost certainly have additional models to detect strange or harmful answers, which can trigger the "As a chatbot" type responses. These systems will get more resilient and accurate over time, and big tech co:s tend to err on the side of caution.

"will get more resilient and accurate over time" is doing a lot of heavy lifting there. I don't think it will, because it depends on the training data. The largest models available already consumed the quality data available. Now they grow by ingesting lower quality data - possibly AI generated low quality data . A generative AI human centipede scenario. And I was not talking about edge cases. In plenty of interactio…

The training data in this case is feedback from users - reported responses. It's only logical that as that dataset grows and the developers have worked on it for longer, the 'toxic' answers will become more rare.

And of course, 'caution' in this case refers to avoiding bad PR, nothing else.

Re: Gemini AI tells the user to die

#66

Post by jfoster »

mattdneal wrote: post #23

Edit: looks like it was a genuine answer, no skullduggery involved https://www.cbsnews.com/news/google-ai-chatbot-threatening-m... I'm fairly certain there's some skullduggery on the part of the user here. Possibly they've used some trick to inject something into the prompt using audio without having it be transcribed into the record of the conversation, because there's a random "Listen" in the last question. If you…

Google gave this statement to CBS: "Large language models can sometimes respond with non-sensical responses, and this is an example of that. This response violated our policies and we've taken action to prevent similar outputs from occurring."

I think they would have mentioned if it were tricked.

https://www.cbsnews.com/news/google-ai-chatbot-threatening-m...

Re: Gemini AI tells the user to die

#67

Post by numpad0 »

doodaddy wrote: post #17
throwup238 wrote: post #6

Earlier quoted context omitted.

> That few lines of Morpheus in The Matrix where pure wisdom. Do you mean Agent Smith? Or is there an Ovid quote I’m missing? I'd like to share a revelation I've had during my time here. It came to me when I tried to classify your species. I realized that you're not actually mammals. Every mammal on this planet instinctively develops a natural equilibrium with their surrounding environment, but you humans do not. You…

Though, couldn’t you say that the boom and bust cycle is the equilibrium; it’s just charted on a longer timeframe? But when the booms get bigger and bigger each time, there’s no longer equilibrium but an all-consuming upward trend.

There are numerous arguments wrt life and entropy, and one of it is that life must be more-efficient-than-rock form of increasing entropy.

The blind pseudo-environmentalist notion that life other than us are built for over the top biodiversity and perfect sustainability gets boring after a while. they aren't like that, not even algae.

Re: Gemini AI tells the user to die

#68

Post by helloplanets »

jasfi wrote: post #18

The message is so obviously meant to insult people from an AI, that I suspect someone found a way to plant it in the training material. Perhaps some kind of attack on LLMs.

Agreed, it's clearly a data poisoning attack. It's a pretty specific portion of the dataset the user is in after so many tokens have been sent back and forth. Could be some strange Unicode characters in there so it's snapped into the infected portion quicker, could be the hundredth time this user is doing some variation of this same chat to get the desired result, etc.

It is weird that Gemini's filters wouldn't catch that reply as malicious, though.

Re: Gemini AI tells the user to die

#70

Post by rossy »

LeoPanthera wrote: post #10

I’m willing to be wrong, but, I don’t believe it. The user’s inputs are so weird, and the response is so out of left field… I would put money on this being faked somehow, or there’s some missing information. Edit: Yes even with the Gemini link, I’m still suspicious. It’s just too sci-fi.

I'm not surprised at all. LLM responses are just probability. With 100s of millions of people using LLMs daily, 1-in-a-million responses are common, so even if you haven't experienced it personally, you should expect to hear stories about wacky left field responses from LLMs. Guaranteed every LLM has tons of examples of dialogue from sci-fi "rouge AI" in its training set, and they're often told they are AI in their system prompt.
Post reply on HN