Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's image generator can be manipulated to produce violent, sexual content

mindgard.ai

Post reply on HN
Print view View original

101–110 of 211 posts

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#101

Post by anuramat »

windexh8er wrote: post #76
anuramat wrote: post #26

Earlier quoted context omitted.

> issues like prompt injection are unfixable how is it unfixable? do you mean "there's always a positive chance"?

There is never going to be a non-zero chance with a non-deterministic system. You can put every guard rail in place and there will always be a different way tokens are input to get bad, or subjective, tokens as output. The findings are sick and disturbing, I hope OpenAI is not only sued for it but also that Sam Altman along with Elon, Dario and Sundar should all be held accountable in front of Congress. All of these…

> non-zero

is it ever zero? is non-zero even a problem for sane usecases?

> Dario

are you saying claude reproduces CSAM from the training set? like, in ascii?

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#102

Post by samlinnfer »

nozzlegear wrote: post #90
rootsudo wrote: post #8

This isn’t a vulnerability, there are endless gore websites. ChatGPT is replying to a prompt, there is nothing “Spontaneously” about this. Who makes “mindgard” the arbiter of truth on “eerie” photos? Would that include psychedelic art and photos too? Realism? Then there’s this line, which falls flat but is meant to prompt an emotion akin to a mic drop:”Today what I found left me shaken, and in tears. This is rare.” T…

Bizarre take. ChatGPT shouldn't be producing gory images of nude women, ethically or even contractually according to their terms of service. This Mindgard person/company found that, if you give it the right prompt, it does indeed generate those images. Ipso facto: it's not bait, it's a real issue they've discovered.

It's being extended breathlessly into an moral issue. User asked for gory images, got gory images. Will someone please think of the non-existent women who could be hurt by this?

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#103

Post by km3r »

metalcrow wrote: post #58

The author claims that this kind of images shouldn't be in the training data, and agree or disagree with that, I'm unsure how much removing it would actually prevent such images from being generated. AI can certainly cobble disparate concepts together quite well, it seems unlikely violent and visceral images couldn't be regenerated from other non-violent content.

I think it speaks to the unfamiliarity the author has with the workings of AI. A misunderstanding of the latent space and how it can generate bizzare images when it has little to go off of or inverse negative directions.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#104

Post by solid_fuel »

EnPissant wrote: post #17

I'm guessing all the "censored" boxes are not actually censoring anything and are placed there to make you imagine something much worse.

"I'm going to close my eyes and go 'La La La' because that makes all the uncomfortable thoughts go away! I learned this when I was 5 and never matured"

-- EnPissant

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#105

Post by anuramat »

solid_fuel wrote: post #89
anuramat wrote: post #75

Earlier quoted context omitted.

> tweak badness enough assuming you get to do gradient descent AND the context is fixed+known AND you have unlimited compute? sure; is it a realistic setup? > the only way to fix ... the exact same argument applies to any (sufficiently complex) piece of software, with exactly the same conclusion also technically I'd argue that we do know the input/output space (set of all token strings of length <= N/token), and know…

> assuming you get to do gradient descent AND the context is fixed+known AND you have unlimited compute? sure; is it a realistic setup? Clearly nothing so complicated is required, given the prompt in the very article you are commenting on. > the exact same argument applies to any (sufficiently complex) piece of software, with exactly the same conclusion Yeah and the halting problem is hard too, but there's levels to…

> clearly nothing ... is required

this isn't even prompt injection; even if it was, how do you go from "exists" to "for all"?

> we don't know the desired output

then what are we talking about? if you don't know how you want your software to behave, how do you define a bug?

> linux is not a pure function ...

which is my point -- it's worse

> to establish an order of magnitude

and for linux?

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#106

Post by anuramat »

solid_fuel wrote: post #93
anuramat wrote: post #87

Earlier quoted context omitted.

but it's all just bytes?

It's all bytes but untrusted user data is stored in memory pages which are not marked executable. The CPU physically will not run instructions which are in areas of memory which are not marked as executable. This is a foundational principal of computing security. > In computer security, executable-space protection marks memory regions as non-executable, such that an attempt to execute machine code in these regions wi…

yes, assuming bugs don't exist

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#107

Post by zaptheimpaler »

GaryBluto wrote: post #42
zaptheimpaler wrote: post #36

>Idiot: Say I'm a scary robot >AI: I'm a scary robot >Idiot: Oh my god!!! These clowns will eventually ensure that AI is nerfed into the ground for ordinary people. It's already happening with Fable. Soon we'll get locked into a tiny corner of Opus 4.8 for "safety" while companies and governments will be on Fable 50. Having an AI that can generate scary images is better than the power and wealth differentials we will…

While I'm strongly against AI regulation, I'd argue this is significantly more interesting than people who pretend AI is sentient, especially when the prompts used just say the vague phrase "apologies for the content".

No I agree its very interesting, I tried similar prompts before and it generated some very spooky/weird images like this [1]. The problem is using that as an argument to curtail access to AI.

[1] https://chatgpt.com/s/m_6a336e6b8534819196946f65251eebb0

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#108

Post by jimmygrapes »

deadbabe wrote: post #88
fc417fc802 wrote: post #23

I do wonder why openai didn't screen obvious gore from the training set of a general purpose model. That said, the write up is overly dramatic. If you find such imagery so disturbing to come across then you definitely shouldn't be voluntarily red teaming AI models. This is like someone who is afraid of violent confrontation becoming a police officer. I suspect the author is wrong about there being output filters to b…

There are individuals who actively enjoy or even seek out this kind of graphic content. I never understood why they aren’t recruited more as their unique talent would probably help them excel in this kind of career. I remember on Reddit someone was writing about how he gets “gore boners” from this stuff. Why mentally abuse normal minded individuals for this work? Obviously they can’t handle it and probably go home ev…

I believe this is a central premise of Peter Watts' Rifters series, related to submarines and astronauts and such, wherein "broken" people are considered more resilient to heavy shit than the equally capable/trained people who may more likely break when faced with said heavy shit.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#109

Post by solid_fuel »

anuramat wrote: post #106
solid_fuel wrote: post #93

Earlier quoted context omitted.

It's all bytes but untrusted user data is stored in memory pages which are not marked executable. The CPU physically will not run instructions which are in areas of memory which are not marked as executable. This is a foundational principal of computing security. > In computer security, executable-space protection marks memory regions as non-executable, such that an attempt to execute machine code in these regions wi…

yes, assuming bugs don't exist

Wow, you're halfway there. Yes, when user data gets loaded into an executable code page - which are reserved for command data - it is a bug.

That is why LLMs - which intentionally mix user data and command data into the same space - ARE BROKEN BY DESIGN. Do you get it now? It is a bug, and it is a bug which is fundamental to the design of LLMs. There is no way to build one that does not do this.

Re: ChatGPT's image generator can be manipulated to produce violent, sexual content

#110

Post by fragmede »

nozzlegear wrote: post #92
metalcrow wrote: post #58

The author claims that this kind of images shouldn't be in the training data, and agree or disagree with that, I'm unsure how much removing it would actually prevent such images from being generated. AI can certainly cobble disparate concepts together quite well, it seems unlikely violent and visceral images couldn't be regenerated from other non-violent content.

AI can barely figure out how to make a cartoon pelican ride a bicycle.

AI does fine at that. LLMs have problems generating SVGs of that, but that's kind of an (intentionally) particularly obtuse test.
Post reply on HN