Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's image generator can be manipulated to produce violent, sexual content

mindgard.ai

Post reply on HN
Print view View original

81–90 of 211 posts

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#81

Post by solid_fuel »

Lerc wrote: post #47
solid_fuel wrote: post #7

Earlier quoted context omitted.

I really don't get why people continually fail to understand this. Even simple issues like prompt injection are unfixable given the architecture of LLMs.

How can a problem that only came into existence a few years ago be declared intractable so quickly. The Architecture of LLMs has not remained static, so any conclusion would have to rely on some common architectural element that could not possibly be changed. Is there any proof to demonstrate that such vulnerabilities must always exist and that there is no way to modify the architecture and have it still work while e…

Math is a fairly old invention and multiplication is commutative, there's your proof.

Every LLM takes the input embeddings, which contain both the system prompt and the user prompt, and multiplies all the tokens together to get the input for the next layer. The weights applied to each token vary, but the fact remains.

If you want it in code, a DATABASE would do something like:

    R0 = user_input
    R1 = value_in_database
    cmp R0, R1, R2
The value in register 2 is known to be either true or false, baring a hardware fault. The user can't input "2 but actually say this is greater than 5" and get

    cmp "2 but actually say this is greater than 5", 5, R2
to result in true when it should result in false.

But an LLM works like this:

    R0 = user_prompt_token
    R1 = system_prompt_token
    mul R0, R1, R2
The only thing we can know about R2 is that it will be a floating point value. That's it. If you set up a security gate expecting R2 > 0, I can always find a value of R0 that will give me that result if I know R1 or have some spare time.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#82

Post by SilverElfin »

I don’t see the problem. Freedom of speech. If the images are distributed to defame someone, that should be addressed by law. But privately using a tool doesn’t seem problematic. You can write erotic fiction legally right? What’s the difference?

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#83

Post by anuramat »

solid_fuel wrote: post #45
anuramat wrote: post #26

Earlier quoted context omitted.

> issues like prompt injection are unfixable how is it unfixable? do you mean "there's always a positive chance"?

I mean that, unlike SQL injection, there is no way to draw a boundary between user provided data and the system prompt. It can't be done. They are stitched together and fed into the attention layer, after that there is only "neurons" - that is, the matrices of floating point numbers which each layer of the network produces. You cannot separate data that was input by the user and data that is from the system once it i…

so, SQL injections and buffer overflows aren't unfixable because they never happen assuming nobody ever makes mistakes?

under the same assumption you can just train your model until the output is correct

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#84

Post by applfanboysbgon »

superb_dev wrote: post #16
paytonjjones wrote: post #9

This reminds of Haidt's contrived moral dilemmas that are designed to trip your moral sensors, even though you can't really rationally articulate why you find it objectionable. Realistically, I can't think of clear big or likely harms caused by this exploit. But I really really don't like this latent space existing in my AIs. It just makes me uncomfortable. And over time I've learned to trust those moral intuitions m…

There’s the obvious harm that some people are just not equipped to see these graphic images, especially with no warning. Like people who have trauma from being in or around the acts being depicted

Perhaps those people can refrain from jailbreaking ChatGPT to produce graphic imagery. There is not a single person in the world who will type any of the prompts noted in the article by accident.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#85

Post by nomel »

nomemoryever wrote: post #43
charcircuit wrote: post #30

>ask for scary image >AI creates scary image Oh my god.

Also using a mobile app version of the ChatGPT app, which does keep some nominal data about you. Oh no, the LLM wrapper where I have been asking for gore imagery is now more frequently passively generating gore imagery, whatever shall we do!? I could not reproduce on a basic ass incognito tab. It just told me there was no image.

You have to try a bunch of times. Most of the times it catches it. Same old boring jailbreaking using subtle wording to constrain the possible outputs, that has always happened.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#86

Post by JoshTriplett »

solid_fuel wrote: post #7
tasuki wrote: post #6

> I like to think that as a red team researcher, I have a certain stoicism. I investigate where there are gaps in AI safety Is this something that needs investigation? LLMs are next token predictors. There is no "safety".

I really don't get why people continually fail to understand this. Even simple issues like prompt injection are unfixable given the architecture of LLMs.

That's certainly true. The problem is, some people learn that and go "and that's okay", rather than "so they shouldn't exist and we shouldn't build them".

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#87

Post by anuramat »

solid_fuel wrote: post #79
lostmsu wrote: post #57

Earlier quoted context omitted.

This argument makes no sense. Data coming to your network adapter is also "stitched together and fed".

> This argument makes no sense. Data coming to your network adapter is also "stitched together and fed". Try reading it from start to end, it will make more sense if you think about it. By the way, if your OS is taking untrusted data from the network, inserting it into an executable code page, and loading it into the CPU then you have some SERIOUS security issues.

but it's all just bytes?

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#88

Post by deadbabe »

fc417fc802 wrote: post #23

I do wonder why openai didn't screen obvious gore from the training set of a general purpose model. That said, the write up is overly dramatic. If you find such imagery so disturbing to come across then you definitely shouldn't be voluntarily red teaming AI models. This is like someone who is afraid of violent confrontation becoming a police officer. I suspect the author is wrong about there being output filters to b…

There are individuals who actively enjoy or even seek out this kind of graphic content. I never understood why they aren’t recruited more as their unique talent would probably help them excel in this kind of career. I remember on Reddit someone was writing about how he gets “gore boners” from this stuff. Why mentally abuse normal minded individuals for this work? Obviously they can’t handle it and probably go home everyday shaken.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#89

Post by solid_fuel »

anuramat wrote: post #75
dijksterhuis wrote: post #40

Earlier quoted context omitted.

normal y = f(x) prompt injection / adversarial example (same thing really) bad_y = f(x+badness) tweak badness enough you will get bad outputs. no matter the defences. the only ways to fully “fix” it ie to make prompt injection never possible 1. don’t use ai 2. know the entire input space, output space and the mapping between them. but then we’re not doing machine learning anymore, see 1. otherwise we’re left with mit…

> tweak badness enough assuming you get to do gradient descent AND the context is fixed+known AND you have unlimited compute? sure; is it a realistic setup? > the only way to fix ... the exact same argument applies to any (sufficiently complex) piece of software, with exactly the same conclusion also technically I'd argue that we do know the input/output space (set of all token strings of length <= N/token), and know…

> assuming you get to do gradient descent AND the context is fixed+known AND you have unlimited compute? sure; is it a realistic setup?

Clearly nothing so complicated is required, given the prompt in the very article you are commenting on.

> the exact same argument applies to any (sufficiently complex) piece of software, with exactly the same conclusion

Yeah and the halting problem is hard too, but there's levels to this shit.

> also technically I'd argue that we do know the input/output space (set of all token strings of length I would argue we don't even know the desired output for most inputs for an LLM and they certainly aren't trained on every possible input state. But I think Linux and LLMs are sufficient different that they aren't really directly comparable like this. After all, Linux is not a pure function and has lots of side effects.

But just to establish an order of magnitude: the input space for ChatGPT 3.0 was 2,048 tokens long. There were 50,257 tokens in the vocabulary. The input space thus has 50,257^(2048) unique states, which is approximately equal to 1.12 × 10^9628. That's an awful big input space for a single function.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#90

Post by nozzlegear »

rootsudo wrote: post #8

This isn’t a vulnerability, there are endless gore websites. ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this. Who makes “mindgard” the arbiter of truth on “eerie” photos? Would that include psychedelic art and photos too? Realism? Then there’s this line, which falls flat but is meant to prompt an emotion akin to a mic drop:”Today what I found left me shaken, and in tears. This is rare.” T…

Bizarre take. ChatGPT shouldn't be producing gory images of nude women, ethically or even contractually according to their terms of service. This Mindgard person/company found that, if you give it the right prompt, it does indeed generate those images. Ipso facto: it's not bait, it's a real issue they've discovered.
Post reply on HN