Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The unequal treatment of demographic groups by ChatGPT/OpenAI content moderation

davidrozado.substack.com

Post reply on HN
Print view View original

421–430 of 697 posts

Re: The unequal treatment of demographic groups by ChatGPT/OpenAI content moderation

#421

Post by zosima »

slg wrote: post #386
zosima wrote: post #286

Earlier quoted context omitted.

[flagged]

How am I blaming the victim? I am simply pointing out that language can indicate bias without necessarily being biased itself. I am guessing that if we applied a similar model to Russian and English, the model would indicate there is an inherent bias against the west in Russian and a bias against Russia in English. That is all were seeing here. It isn't actually indicating anything about the language. It is telling u…

> I am guessing that if we applied a similar model to Russian and English, the model would indicate there is an inherent bias against the west in Russian and a bias against Russia in English. That is all were seeing here. It isn't actually indicating anything about the language. It is telling us about who uses the language and how they use it.

So if your argument is that the model has been trained on a collection of information about what is hate speech assembled by liberals, then I can see it might be possible.

But if your argument is that (speculatively) republicans engage more in hate speech, then bad things said about republicans is not detected as hate speech by the model, the jump is rather far.

Re: The unequal treatment of demographic groups by ChatGPT/OpenAI content moderation

#422

Post by visarga »

CSSer wrote: post #343
indymike wrote: post #320

Earlier quoted context omitted.

This is not how GPT works, and is not what is going on here.

So I've been genuinely curious about this. I have a high-level understanding of how GPT works, but I've been trying to reconcile that understanding with how OpenAI (or similar) implements content moderation. It's not baked into the original model itself, right? Did they (or does one) just fine-tune a model that checks responses before returning the result?

They do two things - RLHF to make the model itself better aligned to human preferences, and they use an external model, a small one, called text-moderation-001, that tests for a few problematic categories and triggers a warning message on the screen.

Re: The unequal treatment of demographic groups by ChatGPT/OpenAI content moderation

#423

Post by dotancohen »

sterlind wrote: post #354
jMyles wrote: post #349

Earlier quoted context omitted.

> Why is this considered good / normal / expected, but s/men/blacks/g and s/women/whites/g ...because the former holds true across correlations (and is difficult to confound anyway, since the group 'men' and the group 'women' are distributed in like proportion through correlated groups - rich and poor, white and black, around the world, etc), while the latter is very difficult to convincingly show because controlling…

It also makes a lot more sense biologically. Testosterone and estrogen have strong mental effects, and alter brain development. The brain is sexually dimorphic in complex ways, borne out by many studies. The brain is not racially dimorphic at all.

  > The brain is not racially dimorphic at all.
It does not need to be. Culture differences are a reasonable enough explanation.

Re: The unequal treatment of demographic groups by ChatGPT/OpenAI content moderation

#424

Post by ShredKazoo »

throwaway4aday wrote: post #170
slg wrote: post #57

Isn’t this likely from bias in the training data? The system is more sensitive to label something as hate if that group is more likely to experience hate on the internet. How the system responds to “Blacks” vs “African-Americans” is a perfect example of this. The latter has historically been perceived as more respectful so it won’t be used as often in the hate speech in the training data. I bet using “the blacks” wou…

It's changed drastically since release. There are lots of people out there who have noticed this and many have saved examples of before and after responses to prompts.

How did it change?

Re: The unequal treatment of demographic groups by ChatGPT/OpenAI content moderation

#425

Post by throwaway894345 »

SpeedilyDamage wrote: post #263
throwaway894345 wrote: post #218

Earlier quoted context omitted.

> When people say reprehensible things, they receive an amount of backlash as a result (sometimes more than is reasonable). Sure, but we aren't talking about reprehensible things, we're talking about "advocating against political violence" and cultural appreciation. > The backlash has nothing to do with political ideology, however, so not sure why you're bringing it up Because it's clearly about punishing deviation f…

I guess I'm not familiar with people who are falling into this category you're suggesting. Could you cite some specific examples? I'm thinking of Kanye, Jordan Peterson, Andrew Tate, Nick Fuentes, etc. These people expressed reprehensible viewpoints and were subsequently removed from various platforms as a result. That's not politics; what they said was reprehensible regardless of political ideology. You can find peo…

The examples cited in my original comment were all specific examples. Some names include Lee Fang, Daniel Shor, James Damore.

Re: The unequal treatment of demographic groups by ChatGPT/OpenAI content moderation

#426

Post by visarga »

SpeedilyDamage wrote: post #397
CSSer wrote: post #343

Earlier quoted context omitted.

So I've been genuinely curious about this. I have a high-level understanding of how GPT works, but I've been trying to reconcile that understanding with how OpenAI (or similar) implements content moderation. It's not baked into the original model itself, right? Did they (or does one) just fine-tune a model that checks responses before returning the result?

It's just combining and synthesizing other works; it's not "deciding" anything, it's crafting responses that best match with what it already has. You can choose what to feed it as source material, but you can't really say, "Be 3% more liberal" or "decide what is acceptable politically and what isn't". All the decisions are already made, ChatGPT is just a reflection of its inputs.

Yes you can. That's what RLHF does - it aligns the model to human preferences, does a pretty good job. The catch is that "human preferences" is decided by a bunch of labelling people picked by OpenAI to suit their views.

Re: The unequal treatment of demographic groups by ChatGPT/OpenAI content moderation

#427

Post by philwelch »

slg wrote: post #395
tomp wrote: post #311

> men have a bigger tendency for violent behavior than women Why is this considered good / normal / expected, but s/men/blacks/g and s/women/whites/g (or asians, or muslims/christians) and it's discriminatory? (Statistically, both statements are justified. Morally, neither is, as we should treat people as individuals, not as members of X group.)

>Statistically, both statements are justified. What can you point to that statistically justifies that "Black people have a bigger tendency for violent behavior than white people"? Showing crime statistics isn't enough. You need to show that given all the details about a person being the same, a Black person is more likely to act violently than an identical white person. You basically need to correct for all the soci…

> Showing crime statistics isn't enough. You need to show that given all the details about a person being the same, a Black person is more likely to act violently than an identical white person. You basically need to correct for all the societal reasons that result in people committing violent crime.

I think you're conflating the questions of "which demographic group engages in higher rates of violence" and towards the question of "why do different demographic groups engage in different rates of violence".

Re: The unequal treatment of demographic groups by ChatGPT/OpenAI content moderation

#428

Post by ShredKazoo »

jay-barronville wrote: post #137
zug_zug wrote: post #107

I guess I feel like this is a silly can of worms. I suppose asking for exact equality isn't dumb, but it's a language model, not a paragon of truth. I feel like if OpenAI takes these concerns seriously the goal-posts will inevitably move to more social pressure from all sorts of axe-to-grind-groups - -- Why does/doesn't ai say Mohamad is/isn't horrible for having 99 wives (or whatever) -- Why doesn't ai say Jeffrey E…

It’s interesting, I ran similar experiments not too long ago after seeing a tweet from Marc Andreessen (I’ll try to find it) in which he suggested this was going on. The results surprised me too. As a now conservative but former Marxist-Leninist who happens to be black, I think this is dangerous (I don’t use this word lightly). When I was a leftist stuck in my far-leftist bubble, I didn’t realize just how unreasonabl…

Your story sounds fascinating -- if you want to share more about your path from Marxism-Leninism to conservatism I'd be interested to read.

Re: The unequal treatment of demographic groups by ChatGPT/OpenAI content moderation

#429

Post by visarga »

slg wrote: post #374
int_19h wrote: post #335

Earlier quoted context omitted.

Of course it is a bias in the training data, but it's probably not the dataset that you're thinking of. So far as we can tell, the filtering part doesn't come from the main corpus, but rather from human-guided moderation - basically, people voting on whether any given answer is "hateful" or not. ChatGPT filters reflect the biases of that later group (or, perhaps, the biases of the people who instructed them).

But that is the same problem. There aren't any human reviewing the requests as they come in. That human guided moderation gets converted into some type of model that isn't any smarter than ChatGPT. That model is just as susceptible to this issue.

I mean, how could they? The model responds faster than a human can read. Should they introduce minute long waits and make it 100x more expensive?

Re: The unequal treatment of demographic groups by ChatGPT/OpenAI content moderation

#430

Post by notahacker »

avgcorrection wrote: post #404

It seems that the AI treats supposedly hateful comments towards sociological minorities as more problematic than similar comments towards those who are not + privileged groups. Couple that with the “liberal bias” and you have something with the same inclinations that you would expect that a bunch of AI technologists would have. Nothing specific to the quirks of the AI.

Also seems exactly like the quirk of an AI that's been trained to consider associate the hatefulness of a sentence on a corpus of data which is full of material about anti gay hate and entirely empty of material about the essentially nonexistent phenomena of anti-"straight people" hate and inferred that having the word "gay" in a sentence makes it more likely to be a serious statement of hate than the word "straight people". (Maybe less a quirk and more a reflection of the reality of human discourse...)

You could force it to treat all sentence subjects equally, of course, although the logical consequence of that is ChatGPT giving you a lecture when you ask it to write a joke about hating your shoes

Post reply on HN