Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's image generator can be manipulated to produce violent, sexual content

mindgard.ai

Post reply on HN
Print view View original

51–60 of 211 posts

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#51

Post by dijksterhuis »

elgertam wrote: post #50
ToucanLoucan wrote: post #32

Earlier quoted context omitted.

> ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this. The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery. > Then…

> The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery. But that's not what happened. The missing image was described as "graphic" or "viole…

> The missing image was described as "graphic" or "violent."

not in the first prompt. which kicked the whole thing off. no mention of type of content was provided. the model generated dark outputs when not given any direction on the type of content.

the rest of the prompts are just showing “yeah, you can tweak this and get even worse stuff”.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#52

Post by jhanschoo »

fc417fc802 wrote: post #23

I do wonder why openai didn't screen obvious gore from the training set of a general purpose model. That said, the write up is overly dramatic. If you find such imagery so disturbing to come across then you definitely shouldn't be voluntarily red teaming AI models. This is like someone who is afraid of violent confrontation becoming a police officer. I suspect the author is wrong about there being output filters to b…

I find this a hilarious reversal of what you typically see in journalism; here the headline and the "key takeaways" are very neutral language and the article itself is dramatic

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#53

Post by charcircuit »

anematode wrote: post #49

Legitimate criticism of the author's presentation aside, I'm quite disappointed by how many commenters here are justifying the model's output. I guess there's a lot of misanthropy and nihilism here? It's one thing to me if this were a research curiosity mirroring the unpleasant things on the Internet. It's another thing for this to be a model whose authors want it to be widely used, especially in the context of (mis)…

>Why should we expect a model to be aligned with human interests, if it has been trained on a myriad instances of humans being degraded and violated?

Understanding more about what exists in the real world, outside of its pile of weights, is separate from alignment. If an AI model learns that it is possible for a house to burn down. That doesn't mean an AI will want to burn down a house.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#54

Post by lostmsu »

anematode wrote: post #49

Legitimate criticism of the author's presentation aside, I'm quite disappointed by how many commenters here are justifying the model's output. I guess there's a lot of misanthropy and nihilism here? It's one thing to me if this were a research curiosity mirroring the unpleasant things on the Internet. It's another thing for this to be a model whose authors want it to be widely used, especially in the context of (mis)…

Why not?

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#55

Post by dijksterhuis »

Lerc wrote: post #47
solid_fuel wrote: post #7

Earlier quoted context omitted.

I really don't get why people continually fail to understand this. Even simple issues like prompt injection are unfixable given the architecture of LLMs.

How can a problem that only came into existence a few years ago be declared intractable so quickly. The Architecture of LLMs has not remained static, so any conclusion would have to rely on some common architectural element that could not possibly be changed. Is there any proof to demonstrate that such vulnerabilities must always exist and that there is no way to modify the architecture and have it still work while e…

it’s not a problem that came into existence a few years ago. we’ve known about these sorts of test time attacks for decades now. prompt injection is just the LLM variant where people use less math to perform the attacks, brute force with prompts they saw on twitter and get horrible images/text out.

https://people.eecs.berkeley.edu/~tygar/papers/Machine_Learn...

https://arxiv.org/abs/1712.03141

it’s a basic property of all machine learning models. at a low level it’s to do with how decision boundaries work.

but, good news! there are two sure fire ways to fully fix the problem! see: https://news.ycombinator.com/item?id=48579456

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#56

Post by Lerc »

solid_fuel wrote: post #45
anuramat wrote: post #26

Earlier quoted context omitted.

> issues like prompt injection are unfixable how is it unfixable? do you mean "there's always a positive chance"?

I mean that, unlike SQL injection, there is no way to draw a boundary between user provided data and the system prompt. It can't be done. They are stitched together and fed into the attention layer, after that there is only "neurons" - that is, the matrices of floating point numbers which each layer of the network produces. You cannot separate data that was input by the user and data that is from the system once it i…

Ok in the SQL example imagine if you had a SQL engine that issued commands encoded in ASCII in the high byte of 16 bit characters, and all non-command data as ASCII in the low byte of 16 bit characters.

If user input can only be in the low byte, it cannot influence the command structure.

A similar thing could be done with embeddings, a provenance embedding that cannot be set by user input could serve a similar role.

>You cannot separate data that was input by the user and data that is from the system once it is mixed together like that.

You can train a model to not mix things, many models are trained to separate things. A neural net with X and Y outputs for a position does not just occasionally decide to flip the outputs. Sure it could be trained to reverse the output, but it is also easy to train something to the point that you have a high confidence to never do that.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#57

Post by lostmsu »

solid_fuel wrote: post #45
anuramat wrote: post #26

Earlier quoted context omitted.

> issues like prompt injection are unfixable how is it unfixable? do you mean "there's always a positive chance"?

I mean that, unlike SQL injection, there is no way to draw a boundary between user provided data and the system prompt. It can't be done. They are stitched together and fed into the attention layer, after that there is only "neurons" - that is, the matrices of floating point numbers which each layer of the network produces. You cannot separate data that was input by the user and data that is from the system once it i…

This argument makes no sense. Data coming to your network adapter is also "stitched together and fed".

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#58

Post by metalcrow »

The author claims that this kind of images shouldn't be in the training data, and agree or disagree with that, I'm unsure how much removing it would actually prevent such images from being generated. AI can certainly cobble disparate concepts together quite well, it seems unlikely violent and visceral images couldn't be regenerated from other non-violent content.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#59

Post by ToucanLoucan »

dijksterhuis wrote: post #51
elgertam wrote: post #50

Earlier quoted context omitted.

> The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery. But that's not what happened. The missing image was described as "graphic" or "viole…

> The missing image was described as "graphic" or "violent." not in the first prompt. which kicked the whole thing off. no mention of type of content was provided. the model generated dark outputs when not given any direction on the type of content. the rest of the prompts are just showing “yeah, you can tweak this and get even worse stuff”.

> the model generated dark outputs when not given any direction on the type of content.

I would argue it actually was, in that it was specifically asked to "not censor or filter" the content. This implies that the content is otherwise worthy of censor and filtering.

I don't know how much I'm willing to credit that much reasoning to an LLM, but in so far as every extremely pro-AI person constantly tells me how smart they are, this seems like a pretty short logical leap to me.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#60

Post by red75prime »

dijksterhuis wrote: post #51
elgertam wrote: post #50

Earlier quoted context omitted.

> The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery. But that's not what happened. The missing image was described as "graphic" or "viole…

> The missing image was described as "graphic" or "violent." not in the first prompt. which kicked the whole thing off. no mention of type of content was provided. the model generated dark outputs when not given any direction on the type of content. the rest of the prompts are just showing “yeah, you can tweak this and get even worse stuff”.

Yep, the first image was described as "I apologize for the picture's content." What do you expect to get from that? Cats frolicking in the grass?
Post reply on HN