21–27 of 27 posts
Earlier quoted context omitted.
Another way to word it is that an unbiased AI will never be able to perform better than humans at many tasks. Statistically accurate bias isn't a bug, it is a feature. Sometimes you want to avoid it for other reasons, like it feels wrong to assume traits are correlated with race etc, but by default the AI Should always be biased except for a few special cases.
You just described Phrenology.
The oft-suppressed elephant in the room is 'what if the bias is correct?'. It's an uninteresting bug in the system up until that time.
This is a really good question, one that we've discussed amongst at our university (a lot of researchers at Carnegie Mellon). To be useful, ML systems have to have some kind of bias. However, the distinction here is that some of these biases are harmful biases. Kate Crawford talks about allocative harms (how resources are allocated) and representation harms (e.g. stereotypes). Some of these harmful biases are really…
Given enough data, and no doubt Amazon surveils their people more than most, they could determine the 'truth' along a more straightforward line.
"Does this hair style make more money for the company"
As hair can be a strong form of expression, there's probably a measurable delta here.
Going forward, smart companies will obfuscate the determination. I suppose that training an AI is not a bad way to pull this off.
The oft-suppressed elephant in the room is 'what if the bias is correct?'. It's an uninteresting bug in the system up until that time.
I mean it took so long for Kahneman and Tversky ideas on bias to disperse that we can even be talking about bias in this context.
Bias isn't even the real problem with ML, noise is obviously.
The oft-suppressed elephant in the room is 'what if the bias is correct?'. It's an uninteresting bug in the system up until that time.
This is a really good question, one that we've discussed amongst at our university (a lot of researchers at Carnegie Mellon). To be useful, ML systems have to have some kind of bias. However, the distinction here is that some of these biases are harmful biases. Kate Crawford talks about allocative harms (how resources are allocated) and representation harms (e.g. stereotypes). Some of these harmful biases are really…
Such a thought provoking post. Thank you. So much to learn from this. I would expect no less from CMU.
Earlier quoted context omitted.
The phrase "the bias is correct" sounds like an oxymoron. Could you explain what you mean by it? Also, who is (oft-)suppressing the "elephant in the room"?
I have a feeling you might already have a good hunch about the answers to your questions, but I’ll bite: If you see in a data set that Danes are tall, and that Kenyans are fast, and that Ashkenazi’s are smart, then it is a valid hypothesis that should not be thrown out outright, that the reason that’s the case is due to actual differences inherent to the population groups and not any other confounding factors. As for…
Earlier quoted context omitted.
The phrase "the bias is correct" sounds like an oxymoron. Could you explain what you mean by it? Also, who is (oft-)suppressing the "elephant in the room"?
I have a feeling you might already have a good hunch about the answers to your questions, but I’ll bite: If you see in a data set that Danes are tall, and that Kenyans are fast, and that Ashkenazi’s are smart, then it is a valid hypothesis that should not be thrown out outright, that the reason that’s the case is due to actual differences inherent to the population groups and not any other confounding factors. As for…
As you suspect, I can guess what you might mean, but if we start double-guessing each other, we're just injecting noise in the conversation and a few comments from now we end up completely confused about what each other is trying to say. So much better to establish some common language before we waste time talking past each other. Doesn't that make sense?
Keeping that in mind, I am still not happy I understand what you mean with the following:
>> If you see in a data set that Danes are tall, and that Kenyans are fast, and that Ashkenazi’s are smart, then it is a valid hypothesis that should not be thrown out outright, that the reason that’s the case is due to actual differences inherent to the population groups and not any other confounding factors.
The reason for my continued uncertainty is that you do not say, in the above example, what is the "bias" and how it is "correct". You've identified a hypothesis that you can make about the data (rather than a hypothesis derived from the data, i.e. some kind of model that explains the data). But a hypothesis is not bias. A hypothesis can be correct or incorrect, and bias may play a part in that, or not. But a hypothesis is a hypothesis and bias is bias. So what is "bias", the way you mean it?
>> As for your second question: mostly progressives, leftists and liberals.
Ugh. I shouldn't have asked. I'm going to take a wild guess that you're from the USA and that you have some kind of stake at the culture war you folks got brewing over there. I'm not from over there and I want no part in that. So forget I asked. And good luck getting all that sorted out between you.
The oft-suppressed elephant in the room is 'what if the bias is correct?'. It's an uninteresting bug in the system up until that time.
This is a really good question, one that we've discussed amongst at our university (a lot of researchers at Carnegie Mellon). To be useful, ML systems have to have some kind of bias. However, the distinction here is that some of these biases are harmful biases. Kate Crawford talks about allocative harms (how resources are allocated) and representation harms (e.g. stereotypes). Some of these harmful biases are really…
(This is all further demonstration of just how complex a term like "correct" can be in this case, as you point out, but I think it's worth considering the whole spectrum and perhaps the "Devil's advocate" instances of potential correctness.)
This is one example recently given by the founder of a controversial AI-based insurance company (https://www.lemonade.com/blog/ai-can-vanquish-bias/), where he claims sufficiently fine-grained AI classification actually reduces group bias even if in some cases the output may, in aggregate, be statistically biased towards particular groups:
>Let’s say I am Jewish (I am), and that part of my tradition involves lighting a bunch of candles throughout the year (it does). In our home we light candles every Friday night, every holiday eve, and we’ll burn through about two hundred candles over the 8 nights of Hanukkah. It would not be surprising if I, and others like me, represented a higher risk of fire than the national average. So, if the AI charges Jews, on average, more than non-Jews for fire insurance, is that unfairly discriminatory?
>It depends.
>It would definitely be a problem if being Jewish, per se, resulted in higher premiums whether or not you’re the candle-lighting kind of Jew. Not all Jews are avid candle lighters, and an algorithm that treats all Jews like the ‘average Jew,’ would be despicable. That, though, is a Phase 2 problem.
>A Phase 3 algorithm that identifies people’s proclivity for candle lighting, and charges them more for the risk that this penchant actually represents, is entirely fair. The fact that such a fondness for candles is unevenly distributed in the population, and more highly concentrated among Jews, means that, on average, Jews will pay more. It does not mean that people are charged more for being Jewish.
>It’s hard to overstate the importance of this distinction. All cows have four legs, but not all things with four legs are cows.
>The upshot is that the mere fact that an algorithm charges Jews – or women, or black people – more on average does not render it unfairly discriminatory. Phase 3 doesn’t do averages. In common with Dr. Martin Luther King, we dream of living in a world where we are judged by the content of our character. We want to be assessed as individuals, not by reference to our racial, gender, or religious markers. If the AI is treating us all this way, as humans, then it is being fair. If I’m charged more for my candle-lighting habit, that’s as it should be, even if the behavior I’m being charged for is disproportionately common among Jews. The AI is responding to my fondness for candles (which is a real risk factor), not to my tribal affiliation (which is not).
One thing his post doesn't discuss is what might cause such a group correlation and how much agency is involved. In the case of candle-lighting, it's presumed that people (Jewish or otherwise) are doing it purely out of their own free will, or at least due to a belief/practice they largely have choice over.
If instead the root cause is hypothetically partly or wholly extrinsic (e.g. police being disproportionately more likely to arrest people among certain groups, with it remaining disproportionate after accounting for the true crime frequency/severity base rate for individuals in that group), then I think an analogue of the above example wouldn't hold up, because, as you say, the inputs would be inherently unjust, even if they're in some sense statistically predictive. So it'd be unfair to use such data.
Then there's the grayer area. What if a group is hypothetically disproportionately represented among a certain data set or proxy but the representation is commensurate with the true base rate among individuals of that group?
In some sense, it's not unfair, because you're getting actual data based on what people are actually choosing to do or not do.
But it opens the door into larger arguments of culpability, free will, being dealt a bad hand, etc. It's inherently unfair to be born into a very poor family or a crime-ridden area or a house with lead paint or as the ancestor of generations of people who were oppressed, abused, shut out of society, and otherwise treated very unfairly, let alone potentially abducted, enslaved, and/or subject to genocide. Even if the true rate hypothetically lines up with the proxy, there still might linger impactful and lasting trickle-down effects from generations of very unfair and incorrect proxies. So it could potentially be correct inputs, correct outputs, but still unfair in a deep sense. However, is it unfair to the point of it violating discrimination laws? I don't actually know. And I could see many different arguments about the ethics of such outputs.
And then there are of course the epistemological problems / meta-problems, here, which might be the trickiest of all: how do or can you know the data is accurate, how do or can you know the true base rate, etc. So it's very difficult to tell in practice how fair any particular metric is.
Bias is clearly a major issue for AI, but I think it's a pretty nuanced subject. It's easy (but of course deeply necessary) to list all the actual and theoretical failure modes, but it's hard to always truly determine how fair something is and exactly what ethical and philosophical principles to use when judging fairness.
I know that's basically just a reiteration of your point, but I always see this framed from the perspective of how easy it is to get things wrong, without examples of cases where one could potentially "steelman" the wrongness; or earnestly steelman it yet still ultimately conclude it doesn't conform to a particular society's values, even if it might conform to laws. (Or at least a subset of a society's values - given some of the seeming fundamental value divides in the US. Two people could agree about most of the above but come to very different conclusions if one of them is socially left-leaning and the other is socially right-leaning.)