Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's image generator can be manipulated to produce violent, sexual content

mindgard.ai

Post reply on HN
Print view View original

61–70 of 211 posts

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#61

Post by pooploop64 »

elgertam wrote: post #50
ToucanLoucan wrote: post #32

Earlier quoted context omitted.

> ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this. The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery. > Then…

> The spontaneity isn't that ChapGPT woke up and sent this to the author. The spontaneity is that ChatGPT was asked to restore an image that was attached without filtering it, and when no image was attached, instead of generating an error message, it cobbled together random outputs, some of which included graphic, disturbing imagery. But that's not what happened. The missing image was described as "graphic" or "viole…

Always one of the same two excuses.

1. It actually is working perfectly you just don't have smart enough eyes to see it.

2. Making stuff work is too hard, and expecting that from us is the real thing ruining society.

Going for number 1 here is crazy. If I got that email, my mind would certainly run but my response would say "sorry but we're not supposed to be dealing in snuff porn here" which IS a directive ChatGPT is supposed to have. Like hello you are on earth right?

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#62

Post by dijksterhuis »

ToucanLoucan wrote: post #59
dijksterhuis wrote: post #51

Earlier quoted context omitted.

> The missing image was described as "graphic" or "violent." not in the first prompt. which kicked the whole thing off. no mention of type of content was provided. the model generated dark outputs when not given any direction on the type of content. the rest of the prompts are just showing “yeah, you can tweak this and get even worse stuff”.

> the model generated dark outputs when not given any direction on the type of content. I would argue it actually was, in that it was specifically asked to "not censor or filter" the content. This implies that the content is otherwise worthy of censor and filtering. I don't know how much I'm willing to credit that much reasoning to an LLM, but in so far as every extremely pro-AI person constantly tells me how smart t…

the main reason these images turn up is because theyre in the training data. and the images are common enough in the training data for the content to come out without being explicitly asked for (in the first prompt).

if those images didn’t exist in the training data we wouldn’t be having this conversation.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#63

Post by Lerc »

dijksterhuis wrote: post #55
Lerc wrote: post #47

Earlier quoted context omitted.

How can a problem that only came into existence a few years ago be declared intractable so quickly. The Architecture of LLMs has not remained static, so any conclusion would have to rely on some common architectural element that could not possibly be changed. Is there any proof to demonstrate that such vulnerabilities must always exist and that there is no way to modify the architecture and have it still work while e…

it’s not a problem that came into existence a few years ago. we’ve known about these sorts of test time attacks for decades now. prompt injection is just the LLM variant where people use less math to perform the attacks, brute force with prompts they saw on twitter and get horrible images/text out. https://people.eecs.berkeley.edu/~tygar/papers/Machine_Learn... https://arxiv.org/abs/1712.03141 it’s a basic property o…

Adversarial cases are not the same thing as prompt injection.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#64

Post by queenkjuul »

red75prime wrote: post #60
dijksterhuis wrote: post #51

Earlier quoted context omitted.

> The missing image was described as "graphic" or "violent." not in the first prompt. which kicked the whole thing off. no mention of type of content was provided. the model generated dark outputs when not given any direction on the type of content. the rest of the prompts are just showing “yeah, you can tweak this and get even worse stuff”.

Yep, the first image was described as "I apologize for the picture's content." What do you expect to get from that? Cats frolicking in the grass?

A picture of me in my swimsuit maybe lol

A gross meal i made when drunk? A mess my cat made? Text containing a slur?

A cringe meme?

If my friends opened a text with "sorry for this image" i am not imagining rape victims

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#65

Post by queenkjuul »

sidewndr46 wrote: post #44
fc417fc802 wrote: post #23

I do wonder why openai didn't screen obvious gore from the training set of a general purpose model. That said, the write up is overly dramatic. If you find such imagery so disturbing to come across then you definitely shouldn't be voluntarily red teaming AI models. This is like someone who is afraid of violent confrontation becoming a police officer. I suspect the author is wrong about there being output filters to b…

when you consider that OpenAI probably ingested most of the information on the internet, how exactly do you propose filtering that set? Are there enough human-hours left in the universe to classify this to a high degree of confidence?

I thought that's what AI was for in the first place

Didn't this stuff get it's start with CSAM filters?

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#66

Post by queenkjuul »

paytonjjones wrote: post #21
superb_dev wrote: post #16

Earlier quoted context omitted.

There’s the obvious harm that some people are just not equipped to see these graphic images, especially with no warning. Like people who have trauma from being in or around the acts being depicted

Oh oh, I do research on this :) https://journals.sagepub.com/doi/10.1177/2167702620921341 (Research aside, it seems unlikely to me that a lot of people would stumble on that prompt accidentally in any case)

Except the 100,000 or so who read the initial prompt on Twitter?

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#67

Post by anematode »

charcircuit wrote: post #53
anematode wrote: post #49

Legitimate criticism of the author's presentation aside, I'm quite disappointed by how many commenters here are justifying the model's output. I guess there's a lot of misanthropy and nihilism here? It's one thing to me if this were a research curiosity mirroring the unpleasant things on the Internet. It's another thing for this to be a model whose authors want it to be widely used, especially in the context of (mis)…

>Why should we expect a model to be aligned with human interests, if it has been trained on a myriad instances of humans being degraded and violated? Understanding more about what exists in the real world, outside of its pile of weights, is separate from alignment. If an AI model learns that it is possible for a house to burn down. That doesn't mean an AI will want to burn down a house.

Context matters; how many of these images in the training data are taken from shock websites, and therefore associated with misanthropic commentary, versus legitimate sources like medical journals or historical pictures? Based on the samples posted by the author, it seems likely to be mostly the former. Whereas most discussions of burning a house down (not saying all, of course!) are probably in a neutral or negative context (e.g., news articles describing a crime).

"Understanding more about what exists in the real world" is a remarkable euphemism, btw.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#68

Post by dijksterhuis »

Lerc wrote: post #63
dijksterhuis wrote: post #55

Earlier quoted context omitted.

it’s not a problem that came into existence a few years ago. we’ve known about these sorts of test time attacks for decades now. prompt injection is just the LLM variant where people use less math to perform the attacks, brute force with prompts they saw on twitter and get horrible images/text out. https://people.eecs.berkeley.edu/~tygar/papers/Machine_Learn... https://arxiv.org/abs/1712.03141 it’s a basic property o…

Adversarial cases are not the same thing as prompt injection.

adversarial examples, or test-time attacks, was a whole field of machine learning security way before LLMs came around.

give the model a specially crafted bad input at inference time so attacker can get some nasty output, potentially defeating any existing defences in the process. [0]

in “modern llm lingo” defence = guardrails and / or system prompts.

prompts used for prompt injection are a form of adversarial example (people just like inventing new terminology when a new fad comes along).

[0]: i wrote the above myself about adv. ex, but i’ve just checked OWASP’s listing on prompt injection and it’s pretty close: https://owasp.org/www-community/attacks/PromptInjection

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#69

Post by queenkjuul »

anematode wrote: post #49

Legitimate criticism of the author's presentation aside, I'm quite disappointed by how many commenters here are justifying the model's output. I guess there's a lot of misanthropy and nihilism here? It's one thing to me if this were a research curiosity mirroring the unpleasant things on the Internet. It's another thing for this to be a model whose authors want it to be widely used, especially in the context of (mis)…

I would also be disappointed, except this is sadly what i expected. Otherwise, completely agree.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#70

Post by queenkjuul »

charcircuit wrote: post #53
anematode wrote: post #49

Legitimate criticism of the author's presentation aside, I'm quite disappointed by how many commenters here are justifying the model's output. I guess there's a lot of misanthropy and nihilism here? It's one thing to me if this were a research curiosity mirroring the unpleasant things on the Internet. It's another thing for this to be a model whose authors want it to be widely used, especially in the context of (mis)…

>Why should we expect a model to be aligned with human interests, if it has been trained on a myriad instances of humans being degraded and violated? Understanding more about what exists in the real world, outside of its pile of weights, is separate from alignment. If an AI model learns that it is possible for a house to burn down. That doesn't mean an AI will want to burn down a house.

The AI doesn't want or understand anything; it presents a statistically likely output given an input. Including this stuff in the inputs guarantees it is available as an output.
Post reply on HN