Biases in AI Systems

>Well I feel that is a really really broad term to just ask for bias without really defining it but a couple off the top of my head are.

I hope this doesn't come off as overtly pedantic, but of course it depends on how you define it. However, even when we DON'T define it, the examples that are used reflect a definition. Fundamentally, the problem is in the continued assumption that data has inherent meaning, rather than being interpreted by human beings in context.[0]

There are two conflicting definitions of bias at play here...

Definition 1 (I believe this is yours): Bias = a purely statistical phenomenon, a situation where one variable is meaningfully predicted by another one. Synonym for correlation.

Definition 2 (I would argue this is mine, the authors, and the colloquial): Biases refer to specific prejudices that are typically unfair. Note...in this definition bias is a synonym for prejudice. They are not making a data argument for that, they are using the term bias to reflect a prejudiced pattern because they understand AI cannot have a 'prejudice' but that societal prejudice can induce biases in Data.

When people use bias to mean definition 2, they are not inherently saying all definition 1 biases are bad. Your examples, under their definition are not biases, they are correlations. The fact that some people use definition 1 does not mean definition 2 is invalid. The authors definition of Bias starts from the idea that a bias is a prejudice, that there are other terms besides bias to describe your examples. Arguing over the definition is distracting you from the point the authors are trying to make...it doesn't actually serve you understanding the article. I say it that way to separate it from anything about this conversation, because I think you are coming at this in good faith and with a sincere point.

No one is forcing AI to ignore that...instead they are saying AI should be able to put that in context of greater societal patterns to be valid. All math, all innovation, all idea exist within a society when they are implemented. It is how they are used that matters. As a thought experiment to your provocative example...ALL pay day loans have a higher default rate...if all mainstream banks .

To give a different example, lets look at the field of educational test and measurement (my current living). I design a test of math ability. I contextualize each question within a game of Cricket. Who do you expect to do well because they have more experience with Cricket? who do you expect to underperform due to the context of the question? ...What am I actually measuring (hint: not just math)? If the data from such tests were inherently 'free from bias' than it wouldn't matter if I asked demographics questions first or last, when in reality - asking demographics questions before a math test lowers Women's scores. Educational test folks constantly ask: what is being measured. They follow it up with a second important question: How is this test, data, and resulting scores going to be used. What does it mean for a test to be fair?[1] When am I trying to measure math skill and I end up measuring gender and poverty instead? If that happens, what can I do with the data? What does the data really mean?

[0] as a fun footnote...this assumption is actually embedded in language used in research, it is why 'Science' fields have tended to stick to third person language while 'social science' and other related fields have largely flipped to first person language.

[1] https://www.ets.org/about/fairness

See my comment here on definition of 'bias'.

https://news.ycombinator.com/item?id=27631529

This is a really good question, one that we've discussed amongst at our university (a lot of researchers at Carnegie Mellon).

To be useful, ML systems have to have some kind of bias. However, the distinction here is that some of these biases are harmful biases. Kate Crawford talks about allocative harms (how resources are allocated) and representation harms (e.g. stereotypes).

Some of these harmful biases are really blatant. For example, labeling Blacks as "Gorillas" is offensive for many reasons.

Some of these biases "correct" but that's due to biases in the data set or society. The ProPublica investigation of recidivism prediction is a good example, where it was more likely to say that Blacks should not be released. However, police are also more likely to arrest Blacks, which naturally leads to this bias. Other examples here include Amazon's resume system that was biased against women (since they used Amazon's hiring practices as ground truth), and image search for "professional hairstyles" that showed White women but "unprofessional hairstyles" that showed Black women.

Other biases are also "correct" but greatly miss the underlying context. For example, a naive AI system might tell you don't go to a certain medical doctor that is a professor, since they have a higher rate of deaths. However, this doctor might also be a doctor of last resort, hence the high mortality rate.

What I'm trying to get to is that even the term "correct" has a lot of subtleties to it. In many cases, figuring out what is "correct" (or ground truth in ML terms) can be a clash of values and world view, and might have different results based on differences in race, gender, age, culture, context, and power.

Rather than being a suppressed topic, in my experience, this is a case of people talking past each other. It's like correlation versus causation (versus plain old connected definitions). It can be true that A and B are correlated, while A doesn't cause B (or neither causes the other), and while their definitions have nothing to do with each other. Like nurse and gender. They're correlated in the US, but making someone a nurse doesn't change their gender, and the definitions have nothing to do with each other. Maybe in some countries the correlation is even flipped!

Recall all the times in stats where an estimator can be an unbiased estimator of a correlation while being a biased estimator of a causal effect.

So you get some people saying it (the correlation) is correct and other people saying it (the causal effect) is incorrect. Both are right! To stop talking past each other, they need to talk about bias with respect to the correlation or bias with respect to the causal effect in this particular direction.

But what frustrates me is when the correlation side uses the (true) correlation to argue against a system being biased with regards to something else (w.r.t. a definition or w.r.t. a causal effect or w.r.t. a literal translation or w.r.t. some more complicated aspect of the system), and that harms are okay because the bias is a correct bias.

We need to work on our terminology so that we can stop talking past each other. It doesn't help that our models have weird biases in absurdly complex function spaces, but we have to progress beyond a first-stats-course one-size-fits-all definition of bias.

I have a feeling you might already have a good hunch about the answers to your questions, but I’ll bite:

If you see in a data set that Danes are tall, and that Kenyans are fast, and that Ashkenazi’s are smart, then it is a valid hypothesis that should not be thrown out outright, that the reason that’s the case is due to actual differences inherent to the population groups and not any other confounding factors.

As for your second question: mostly progressives, leftists and liberals.