21–30 of 92 posts
This is an extraordinarily bold claim. I'd be quite interested in how peoples' responses to the article changed if this was the lead.
Coauthor here. Some of the press articles about our work didn't have a lot of nuance (unsurprisingly), but in the paper we're careful about what we say, what we don't say, and what the implications are. Happy to engage in informed discussion :)
Can you provide an example of how this bias might play out in a human-AI interaction?
- An AI correctly infers (simply by reading text) that a physicist is male and a nurse is female.
- An AI correctly infers the gender of humans with androgyonous names.
- An AI infers insects are unpleasant and flowers are pleasant to humans.
- An AI also infers that African American names are more likely to be associated with unpleasantness than European names.
[edit: to those who dislike this comment, can you tell me what you object to? Which of my concrete examples is not in the paper?]
Be careful with the word truth. You don't know what it means.
Earlier quoted context omitted.
I think your questions would be answered by reading the article. Particularly: "In AI and machine learning, bias refers generally to prior information, a necessary prerequisite for intelligent action (4). Yet bias can be problematic where such information is derived from aspects of human culture known to lead to harmful behavior. Here, we will call such biases “stereotyped” and actions taken on their basis “prejudice…
The usual definition of bias in ML papers is E[theta_estimator - theta]. That is explicitly a systematically wrong prediction. In any case, the paper suggests that this "bias" or "prejudice" is better described as "truths I don't like". I'm asking if the author knows of any cases where they are actually not truthful. The paper does not suggest any, but maybe there are some?
So if 34% of doctors are female, then you would expect 34% of doctors in news or Wikipedia articles to be female. Even if the articles are completely unbiased and the writers have no stereotypes whatsoever. And so the word vector would naturally label "doctor" something like "66% likely to occur in a male context".
And in fact this paper confirms that. Figure 1 shows that the word vectors are highly predictive of the actual gender distribution of various occupations. Probably much more accurate than most people would be. So it's not mindlessly absorbing human stereotypes. It's learning reality's stereotypes.
This result is completely expected and desirable. What makes word vectors so powerful is how they can learn complicated correlations between words and their contexts. The famous example is how it learns that "Queen" is the female equivalent of "King". Which is a gender stereotype as well. If it wasn't able to learn that doctors were a bit more likely to be male, that would be more surprising.
> That is, before providing an explicit or institutional explanation for why individuals make prejudiced decisions, one must show that it was not a simple outcome of unthinking reproduction of statistical regularities absorbed with language. This is an extraordinarily bold claim. I'd be quite interested in how peoples' responses to the article changed if this was the lead.
Very bold. The full quote is worth reproducing:
> Our results also suggest a null hypothesis for explaining origins of prejudicial behavior in humans, namely, the implicit transmission of ingroup/outgroup identity information through language. That is, before providing an explicit or institutional explanation for why individuals make prejudiced decisions, one must show that it was not a simple outcome of unthinking reproduction of statistical regularities absorbed with language.
I'm reminded of Parable of the Polygons[0] which illustrates Shelling's model of segregation, showing how quite small initial biases can be amplified and result in very large segregation.
It would be seem very sad if tribalism in all its forms is simply an emergent behaviour, a result of a random fluctuation (e.g. one or two racist individuals) causing a chain reaction throughout society, where biases become gradually amplified even if most individuals are, themselves, generally well-meaning. How do we escape from that?
Be careful with the word truth. You don't know what it means.
> That is, before providing an explicit or institutional explanation for why individuals make prejudiced decisions, one must show that it was not a simple outcome of unthinking reproduction of statistical regularities absorbed with language. This is an extraordinarily bold claim. I'd be quite interested in how peoples' responses to the article changed if this was the lead.
> This is an extraordinarily bold claim. Very bold. The full quote is worth reproducing: > Our results also suggest a null hypothesis for explaining origins of prejudicial behavior in humans, namely, the implicit transmission of ingroup/outgroup identity information through language. That is, before providing an explicit or institutional explanation for why individuals make prejudiced decisions, one must show that it…
Sadly, that seems to be the case.
I wouldn't even turn racism into a "special case" here. Humans are capable of dividing themselves into ingroups and outgroups over everything, no matter how trivial. I suspect the segregation process will occur with all in/outgroup divisions. Separations along the race and gender lines are particularly prevalent because those are the most obvious, noticeable differentiators between people.
Coauthor here. Some of the press articles about our work didn't have a lot of nuance (unsurprisingly), but in the paper we're careful about what we say, what we don't say, and what the implications are. Happy to engage in informed discussion :)
Do you have any evidence that this effect results in machines making systematically wrong inferences? Near as I can tell, your paper shows that these "biases" result in significantly more accurate predictions. For example, Fig 1 shows that a machine trained on human language can accurately predict the % female of many professions. Fig 2 shows the machine can accurately predict the gender of humans. Normally I'd expec…
From what I understand, the fear surrounding embedding human stereotypes into ML systems is that the stereotypes will get reinforced. In some way or form, there will be less equality of opportunity in the future than exists today, because machines will make decisions that humans are currently making. Societal norms evolve over time, yet code can become locked in place.
Is your takeaway from this paper that we, as the creators of intelligent machines, should allow them to continue to making "positively" right assumptions simply because that's the way we, as humans, have always done them? Is "positively" right, in your opinion, in all cases equivalent to "normatively" right?
Earlier quoted context omitted.
I think your questions would be answered by reading the article. Particularly: "In AI and machine learning, bias refers generally to prior information, a necessary prerequisite for intelligent action (4). Yet bias can be problematic where such information is derived from aspects of human culture known to lead to harmful behavior. Here, we will call such biases “stereotyped” and actions taken on their basis “prejudice…
The usual definition of bias in ML papers is E[theta_estimator - theta]. That is explicitly a systematically wrong prediction. In any case, the paper suggests that this "bias" or "prejudice" is better described as "truths I don't like". I'm asking if the author knows of any cases where they are actually not truthful. The paper does not suggest any, but maybe there are some?
Think for example of an inductive bias. If I see a couple of white swans, I may conclude that all swans are white, and we all know this is wrong. Similarly, I may conclude the sun rises everyday, and for all practical purposes this is correct. This kind of bias is neither wrong nor right, but, in the words of the article "a necessary prerequisite for intelligent action", because no induction/generalization would be possible without it.
There are undoubtedly examples where the prejudiced kind of biases lead to both truthful and untruthful predictions, but that seems beside the point, which is to design a system with the biases you want, and without the ones you don't.