Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grok Sexual Images Draw Rebuke, France Flags Content as Illegal

finance.yahoo.com

Post reply on HN
Print view View original

91–100 of 103 posts

Re: Grok Sexual Images Draw Rebuke, France Flags Content as Illegal

#91

Post by tukarsdev »

chrisjj wrote: post #78
johneth wrote: post #76

Earlier quoted context omitted.

It makes sexual images of real people without their consent. That's what's breaking the law.

Is an image of someone wearing only a bikini seriously claimed to be sexual here? Not by this article, for sure. "The service prohibits pornography involving real people’s likenesses and sexual content involving minors, which is illegal to create or distribute. Still, users have prompted Grok to digitally remove clothing from photos — mostly of women — so the subjects appeared to be wearing only underwear or bikinis.…

Removing people's clothes without their consent is assault, it doesn't matter if, in another setting, where they did consent to it, it would be fine. It obviously is sexual if you look at the intent of people doing it. Not the clothing itself.

Re: Grok Sexual Images Draw Rebuke, France Flags Content as Illegal

#92

Post by judahmeek »

akutlay wrote: post #2

It seems X's Grok became the first large LLM provider to weaken the content moderation rules. If people don't react enough, we will likely lose the first line of defense for keeping AI safe for anyone. Large providers need to act responsibly as the barrier of entry is practically 0.

whether it was the "first" definitely depends on your standards & focus: https://cloudsecurityalliance.org/blog/2025/02/19/deepseek-r...

Re: Grok Sexual Images Draw Rebuke, France Flags Content as Illegal

#93

Post by chrisjj »

tukarsdev wrote: post #91
chrisjj wrote: post #78

Earlier quoted context omitted.

Is an image of someone wearing only a bikini seriously claimed to be sexual here? Not by this article, for sure. "The service prohibits pornography involving real people’s likenesses and sexual content involving minors, which is illegal to create or distribute. Still, users have prompted Grok to digitally remove clothing from photos — mostly of women — so the subjects appeared to be wearing only underwear or bikinis.…

Removing people's clothes without their consent is assault, it doesn't matter if, in another setting, where they did consent to it , it would be fine. It obviously is sexual if you look at the intent of people doing it. Not the clothing itself.

> Removing people's clothes without their consent is assault

Didn't you know? Grok does not actually remove people's clothes. Instead it pastes from photos of /other people who are already naked/.

Re: Grok Sexual Images Draw Rebuke, France Flags Content as Illegal

#94

Post by chrisjj »

krapp wrote: post #79
ben_w wrote: post #71

Earlier quoted context omitted.

I am uncertain about this. Yes, GenAI content is cheap. But a business whose output is identical to everyone else's, because everyone is using the same models to solve the same problems, has no USP and no signal to customers to say why they're different. The meme a while back about OpenAI having no moat? That's just as true for businesses depending on any public AI tool. If you can't find something that AI fails at,…

Most businesses don't compete on difference - most competitors are virtually indistinguishable from one another. Rather they tend to compete on brand identity and loyalty. I think businesses assume the output of AI can be the same as with their current workflow, just with the benefit of cutting their workforce, so all upside and no downside. I also suspect that a lot of businesses (at least the biggest ones) are look…

> Most businesses don't compete on difference - ... Rather they tend to compete on brand identity and loyalty.

Without a difference, brand identity and loyalty are impossible to build.

Re: Grok Sexual Images Draw Rebuke, France Flags Content as Illegal

#95

Post by [deleted] »

ben_w wrote: post #82
chrisjj wrote: post #80

Earlier quoted context omitted.

If we're talking about undressing, there is no aggregate. Some people want something; others want them not to have it. Simple. What the former want is not illegal. So the fact they are a minority is irrelevent. Minorites have rights too. If we're talking about genuine CSAM, that very different and not even limited to undressing.

> If we're talking about genuine CSAM, that very different and not even limited to undressing. Why would you think I was talking about anything else? Also, "subset" != "very different" > What the former want is not illegal. So the fact they are a minority is irrelevent. Minorites have rights too. This is newsworthy because non-consensual undressing of images of a minor, even by an AI, already passes the requisite thr…

[deleted]

Re: Grok Sexual Images Draw Rebuke, France Flags Content as Illegal

#96

Post by chrisjj »

ben_w wrote: post #82
chrisjj wrote: post #80

Earlier quoted context omitted.

If we're talking about undressing, there is no aggregate. Some people want something; others want them not to have it. Simple. What the former want is not illegal. So the fact they are a minority is irrelevent. Minorites have rights too. If we're talking about genuine CSAM, that very different and not even limited to undressing.

> If we're talking about genuine CSAM, that very different and not even limited to undressing. Why would you think I was talking about anything else? Also, "subset" != "very different" > What the former want is not illegal. So the fact they are a minority is irrelevent. Minorites have rights too. This is newsworthy because non-consensual undressing of images of a minor, even by an AI, already passes the requisite thr…

> Why would you think I was talking about anything else?

Because this thread shows CSAM confused with other, e.g. simple child pornography.

And even the source of the quote isn't helping. Clicking its https://www.iwf.org.uk/ "Find out why we use the term ‘child sexual abuse’ instead of ‘child pornography’." gives 403 - Forbidden: Access is denied.

Fortunately a good explanation of the difference can be found here: https://www.thorn.org/blog/ai-generated-child-sexual-abuse-t...

> This is newsworthy because non-consensual undressing of images of a minor, even by an AI

That's not the usage in question. The usage is "generate realistic pictures of undressed minors". Undressing images of real people is prohibited.

Re: Grok Sexual Images Draw Rebuke, France Flags Content as Illegal

#97

Post by [deleted] »

lokar wrote: post #20
ls612 wrote: post #18

Earlier quoted context omitted.

Should photoshop be outlawed? What about MS Paint? Both of them I’m pretty sure are capable of creating this stuff. Also, lets test your commitment to consistency on this matter. In most jurisdictions possession and creation of CSAM is a strict liability crime, so do you support prosecuting whatever journalist demonstrated this capability to the maximum extent of the law? Or are you only in favor of protecting childr…

Photoshop is fine, running a business where you produce CSAM for people with photoshop is not. And this has been very clear for a while now. I did not see the details of what happened, but if someone did in fact take a photo of a real child they had no connection to and caused the images to be created, then yes, they should be investigated, and if the prosecutor thinks they can get a conviction they should be charged…

[deleted]

Re: Grok Sexual Images Draw Rebuke, France Flags Content as Illegal

#98

Post by chrisjj »

lokar wrote: post #20
ls612 wrote: post #18

Earlier quoted context omitted.

Should photoshop be outlawed? What about MS Paint? Both of them I’m pretty sure are capable of creating this stuff. Also, lets test your commitment to consistency on this matter. In most jurisdictions possession and creation of CSAM is a strict liability crime, so do you support prosecuting whatever journalist demonstrated this capability to the maximum extent of the law? Or are you only in favor of protecting childr…

Photoshop is fine, running a business where you produce CSAM for people with photoshop is not. And this has been very clear for a while now. I did not see the details of what happened, but if someone did in fact take a photo of a real child they had no connection to and caused the images to be created, then yes, they should be investigated, and if the prosecutor thinks they can get a conviction they should be charged…

> Photoshop is fine, running a business where you produce CSAM for people with photoshop is not.

The law disagrees - at least in UK. CSAM is illegal regardless of tool used.

> I did not see the details of what happened, but if someone did in fact take a photo of a real child they had no connection to and caused the images to be created

The article makes no report that happened. And it does report that is prohibited by the tool in question. But it does then quote a child safety advocate saying tools should not be allowed to "generate this material", so is misleading in the extreme.

Re: Grok Sexual Images Draw Rebuke, France Flags Content as Illegal

#99

Post by chrisjj »

zajio1am wrote: post #51
lokar wrote: post #20

Earlier quoted context omitted.

Photoshop is fine, running a business where you produce CSAM for people with photoshop is not. And this has been very clear for a while now. I did not see the details of what happened, but if someone did in fact take a photo of a real child they had no connection to and caused the images to be created, then yes, they should be investigated, and if the prosecutor thinks they can get a conviction they should be charged…

> Photoshop is fine, running a business where you produce CSAM for people with photoshop is not. And this has been very clear for a while now. What if Photoshop is provided as a web service? This is analogous to running image generation as a service. In both cases provider takes input from the user (in one case textual description, on the other case sequence of mouse events) and generates and image with an automated…

> Note that in this case using them for producing CSAM

There's no such report in this article.

Re: Grok Sexual Images Draw Rebuke, France Flags Content as Illegal

#100

Post by chrisjj »

ls612 wrote: post #39
lokar wrote: post #32

Earlier quoted context omitted.

You often see (appropriately, IMO) a certain amount of discretion wrt prosecution when things are changing quickly. I doubt anyone will go to jail over this. What (I think) should happen is state or federal law enforcement need to make it very clear to Xai (and the others) that this is unacceptable, and that if it keep happening, and you are not showing that you are fixing it (even if that means some degradation in t…

When the GP said “not possible” they were referring to the strict letter of the law that I was, not to your lower standard of “make a good effort to fix it”. Law is not code because that gives the lawgivers discretion to exercise power arbitrarily while convincing the citizens that they live under the “rule of law”. At least the Chinese for all their faults don’t bother with the pretense.

> When the GP said “not possible” they were referring to the strict letter of the law that I was

I, the GP, was referring to what I quoted:

“AI products must be tested rigorously before they go to market to ensure they do not have the capability to generate this material [CSAM],” and I agree this is in effect the law at least here in UK.

Post reply on HN