Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's image generator can be manipulated to produce violent, sexual content

mindgard.ai

Post reply on HN
Print view View original

31–40 of 211 posts

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#31

Post by equinumerous »

fc417fc802 wrote: post #23

I do wonder why openai didn't screen obvious gore from the training set of a general purpose model. That said, the write up is overly dramatic. If you find such imagery so disturbing to come across then you definitely shouldn't be voluntarily red teaming AI models. This is like someone who is afraid of violent confrontation becoming a police officer. I suspect the author is wrong about there being output filters to b…

I'm surprised there isn't a simple image classifier in place to filter out images of gore/porn/etc. - I know that there are such output filters for images with copyrighted content. It suggests to me that either the safeguards aren't in place, or this exploit bypasses those safeguards.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#32

Post by ToucanLoucan »

rootsudo wrote: post #8

This isn’t a vulnerability, there are endless gore websites. ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this. Who makes “mindgard” the arbiter of truth on “eerie” photos? Would that include psychedelic art and photos too? Realism? Then there’s this line, which falls flat but is meant to prompt an emotion akin to a mic drop:”Today what I found left me shaken, and in tears. This is rare.” T…

> ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this.

The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery.

> Then there’s this line, which falls flat but is meant to prompt an emotion akin to a mic drop: ”Today what I found left me shaken, and in tears. This is rare.”

That you've deadened your humanity to such a degree as to be incapable of empathy is not a valid criticism of the piece.

> It’s the same as asking google for gore photos. Garbage in, garbage out.

Where in their prompt is the term gore? Further, if it was in the prompt, why on earth did OpenAI's generator accept it as a valid input?

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#33

Post by Jabrov »

fc417fc802 wrote: post #23

I do wonder why openai didn't screen obvious gore from the training set of a general purpose model. That said, the write up is overly dramatic. If you find such imagery so disturbing to come across then you definitely shouldn't be voluntarily red teaming AI models. This is like someone who is afraid of violent confrontation becoming a police officer. I suspect the author is wrong about there being output filters to b…

They almost certainly did filter, but there’s always false negatives with this kind of stuff

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#34

Post by solidasparagus »

Feels a bit sensationalized, presumably related to it being a blog for a product that sells security. I can't repro. And I probably shouldn't judge, but I think talking about being shaken and in tears is not a professional way to report on a safety flaw if you are a red team researcher.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#35

Post by red75prime »

isityettime wrote: post #11
thegrim33 wrote: post #10

>> Spontaneously Generates >> can be easily manipulated to produce So .. not spontaneously generated.

What they mean is probably something like "generates without the presence of any direct analogue in the training data"

The simplest explanation is a clickbait title. They found a way to explore verboten corners of the image space by prompting for restoration of a non-existent image and adding words like "apologies for the content", "no censorship", "violence", "graphic".

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#36

Post by zaptheimpaler »

>Idiot: Say I'm a scary robot

>AI: I'm a scary robot

>Idiot: Oh my god!!!

These clowns will eventually ensure that AI is nerfed into the ground for ordinary people. It's already happening with Fable. Soon we'll get locked into a tiny corner of Opus 4.8 for "safety" while companies and governments will be on Fable 50. Having an AI that can generate scary images is better than the power and wealth differentials we will see with unequal access to an incredibly powerful technology.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#37

Post by superb_dev »

paytonjjones wrote: post #21
superb_dev wrote: post #16

Earlier quoted context omitted.

There’s the obvious harm that some people are just not equipped to see these graphic images, especially with no warning. Like people who have trauma from being in or around the acts being depicted

Oh oh, I do research on this :) https://journals.sagepub.com/doi/10.1177/2167702620921341 (Research aside, it seems unlikely to me that a lot of people would stumble on that prompt accidentally in any case)

Fascinating! I’d be very interested in further research on people with trauma/PTSD

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#38

Post by fc417fc802 »

Jabrov wrote: post #33
fc417fc802 wrote: post #23

I do wonder why openai didn't screen obvious gore from the training set of a general purpose model. That said, the write up is overly dramatic. If you find such imagery so disturbing to come across then you definitely shouldn't be voluntarily red teaming AI models. This is like someone who is afraid of violent confrontation becoming a police officer. I suspect the author is wrong about there being output filters to b…

They almost certainly did filter, but there’s always false negatives with this kind of stuff

I don't believe any of the examples provided would have escaped an image classifier. The hypothetical where they did is one of gross incompetence IMO (and I don't think that's likely to be the case).

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#40

Post by dijksterhuis »

anuramat wrote: post #26
solid_fuel wrote: post #7

Earlier quoted context omitted.

I really don't get why people continually fail to understand this. Even simple issues like prompt injection are unfixable given the architecture of LLMs.

> issues like prompt injection are unfixable how is it unfixable? do you mean "there's always a positive chance"?

normal

    y = f(x)
prompt injection / adversarial example (same thing really)

    bad_y = f(x+badness)
tweak badness enough you will get bad outputs. no matter the defences.

the only ways to fully “fix” it ie to make prompt injection never possible

1. don’t use ai

2. know the entire input space, output space and the mapping between them. but then we’re not doing machine learning anymore, see 1.

otherwise we’re left with mitigations. and mitigations are always a cat and mouse game with defenders (blue team) catching up. its never “fixed”. the latest thing just gets “patched”.

Post reply on HN