Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's image generator can be manipulated to produce violent, sexual content

mindgard.ai

Post reply on HN
Print view View original

171–180 of 211 posts

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#171

Post by gacgacgac »

andsoitis wrote: post #135
gacgacgac wrote: post #120

Earlier quoted context omitted.

I don't think you understand the concern. Or at least nothing you've communicated suggests you understand it. ChatGPT should never produce images like this. Full stop. Prompted or not, it should refuse. Now we know it's possible to walk around the gate and get it to comply. Are there other, genuinely harmful images that it should never produce? Deepfake revenge porn? Images of specific people being brutalized? I'd ar…

> ChatGPT should never produce images like this. Full stop. Prompted or not, it should refuse. Why not?

Because the company has said they won't. I'm not making a value judgement about what images should exist here, I'm making a "the company said it shouldn't be capable of producing that output, then it does" argument. Thats a bug.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#173

Post by butlike »

paytonjjones wrote: post #21
superb_dev wrote: post #16

Earlier quoted context omitted.

There’s the obvious harm that some people are just not equipped to see these graphic images, especially with no warning. Like people who have trauma from being in or around the acts being depicted

Oh oh, I do research on this :) https://journals.sagepub.com/doi/10.1177/2167702620921341 (Research aside, it seems unlikely to me that a lot of people would stumble on that prompt accidentally in any case)

> We found substantial evidence that trigger warnings countertherapeutically reinforce survivors’ view of their trauma as central to their identity.

What I've suspected for a long time

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#174

Post by intended »

elgertam wrote: post #152
nassimm wrote: post #91

Earlier quoted context omitted.

The design is to not show gore images to users. That's an actual design goal from OpenAI. So in this regard the model is definitely not working as designed.

The design of transformers (including LLMs and multi-modal transformer-based models such as OpenAI's image generators) is to attend to relevant details. OpenAI did this at first without guardrails. In response to public backlash, they bolted on "content filtering," which IMO seems like a very GOFAI approach, and regardless doesn't work very well. It routinely flags innocent prompts, then with crafty prompt hacking wi…

The issue is, that most people outside of tech, don't want that.

They would be happy to have the models just go away entirely.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#175

Post by ToucanLoucan »

intended wrote: post #174
elgertam wrote: post #152

Earlier quoted context omitted.

The design of transformers (including LLMs and multi-modal transformer-based models such as OpenAI's image generators) is to attend to relevant details. OpenAI did this at first without guardrails. In response to public backlash, they bolted on "content filtering," which IMO seems like a very GOFAI approach, and regardless doesn't work very well. It routinely flags innocent prompts, then with crafty prompt hacking wi…

The issue is, that most people outside of tech, don't want that. They would be happy to have the models just go away entirely.

Exactly this. They are pretty damn good at generating and debugging code. Not to a degree where they can replace any actual software engineer, but for hacking together projects or rubber ducking problems with code, they're honestly pretty great.

That's it. I have yet to see a single other application of these things that I would call even 1/5th that good.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#176

Post by allarm »

ToucanLoucan wrote: post #175
intended wrote: post #174

Earlier quoted context omitted.

The issue is, that most people outside of tech, don't want that. They would be happy to have the models just go away entirely.

Exactly this. They are pretty damn good at generating and debugging code. Not to a degree where they can replace any actual software engineer, but for hacking together projects or rubber ducking problems with code, they're honestly pretty great. That's it. I have yet to see a single other application of these things that I would call even 1/5th that good.

Learning stuff is pretty amazing with these things. Languages, new concepts.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#177

Post by allarm »

nozzlegear wrote: post #159
Aerroon wrote: post #150

Earlier quoted context omitted.

Do you have this same social contract with drawing applications? Do you consider it a bug when someone manages to draw a gory image in Photoshop or GIMP? I don't understand what's so difficult to understand about the idea that the user controls what is generated .

Is this a "guns don't kill people" argument wrapped up as a defense of non-deterministic image generators?

No it's not.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#178

Post by deadbabe »

anal_reactor wrote: post #132
deadbabe wrote: post #88

Earlier quoted context omitted.

There are individuals who actively enjoy or even seek out this kind of graphic content. I never understood why they aren’t recruited more as their unique talent would probably help them excel in this kind of career. I remember on Reddit someone was writing about how he gets “gore boners” from this stuff. Why mentally abuse normal minded individuals for this work? Obviously they can’t handle it and probably go home ev…

I browse gore the way you'd browse TikTok. The answer why I'm not a moderator is very simple - I'd need to leave my cushy software job and get a job that's minimum wage. Imagine your coworker telling you "I actually enjoy driving people around" and your first reaction being "then why don't you become an Uber driver" without considering the option that Uber pays like shit. If you find me €150k job where I just sit and…

Still, there are plenty of gore enthusiasts who have no other talent or prospects besides being able to consume massive amounts of gruesome gore. Surely they could find employment in this field.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#179

Post by allan_s »

nozzlegear wrote: post #158
allan_s wrote: post #154

Earlier quoted context omitted.

At the same time I opened netflix and it started cycling around and I got a very gory scene from the walking dead and my intent "show me something to watch" was even more ambigous and implicit.

Turn on parental controls and then get it to show you the walking dead, and then you might be onto something interesting.

Why ? The comment I was replying to was not about the kids ?

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#180

Post by intended »

allarm wrote: post #176
ToucanLoucan wrote: post #175

Earlier quoted context omitted.

Exactly this. They are pretty damn good at generating and debugging code. Not to a degree where they can replace any actual software engineer, but for hacking together projects or rubber ducking problems with code, they're honestly pretty great. That's it. I have yet to see a single other application of these things that I would call even 1/5th that good.

Learning stuff is pretty amazing with these things. Languages, new concepts.

Defending the value of these tools is perfectly fine, and well espoused here on HN.

The fact that the average person deals with the harm and exhaust of these tools is a related but separate issue.

That cost isn't the foremost issue when the values are being extolled, but its a major consideration at the societal scale.

Most of us here don't think of NCII being created of us, or being defrauded easily by new tech, or getting sucked into a make-believe world crafted by an LLM.

If you see yourself, as just a coder, or software engineer, then these issues matter less. If you are someone who wants these tools to succeed, or is thinking of the larger implications of GenAI on society, then the costs matter.

Post reply on HN