Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Bias

propublica.org

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Machine Bias

#31

Post by escape_goat »

zyxley wrote: post #27
yummyfajitas wrote: post #20

Earlier quoted context omitted.

None of these things are evidence of bias. The algorithm is biased if it's giving the wrong score due to race or redundantly encoded race. To show that the algorithm is biased, you need to show that (score, race) pairs are more predictive than (score, ) singletons. Line [36] and [46] both attempt to address this question. The only one of these which is statistically significant is "race_factorOther:score_factorHigh"…

> but do not show bias. At best they show disparate impact I have no interest in playing but-what-does-the-exact-dictionary-definition-say semantics games.

> I have no interest in playing but-what-does-the-exact-dictionary-definition-say semantics games.

That's certainly reasonable, but actually you're responding to a statistical argument about a statistical study, made by a statistician.

It would seem tendentious to argue that these numbers are unconnected to systemic social bias, but he is not making such an argument.

Re: Machine Bias

#33

Post by yummyfajitas »

TheCoelacanth wrote: post #28
yummyfajitas wrote: post #20

Earlier quoted context omitted.

None of these things are evidence of bias. The algorithm is biased if it's giving the wrong score due to race or redundantly encoded race. To show that the algorithm is biased, you need to show that (score, race) pairs are more predictive than (score, ) singletons. Line [36] and [46] both attempt to address this question. The only one of these which is statistically significant is "race_factorOther:score_factorHigh"…

It is consistently giving incorrectly low scores to white subjects and consistently giving incorrectly high scores to black subjects. That is clearly bias, at least in the colloquial sense.

The degree to which it does this cannot be distinguished from random chance (p > 0.05).

If the predictor were biased then you could build a more accurate score based on both the original scores and race_factorBlack:score_factorHigh (and other interaction terms). I.e. you'd be building a new bias in to cancel the old bias, leaving an accurate predictor.

Their analysis doesn't show that this is possible.

Re: Machine Bias

#34

Post by yummyfajitas »

jdp23 wrote: post #29
yummyfajitas wrote: post #20

Earlier quoted context omitted.

None of these things are evidence of bias. The algorithm is biased if it's giving the wrong score due to race or redundantly encoded race. To show that the algorithm is biased, you need to show that (score, race) pairs are more predictive than (score, ) singletons. Line [36] and [46] both attempt to address this question. The only one of these which is statistically significant is "race_factorOther:score_factorHigh"…

"Oh sure, this algorithm is much more likely to have false positives on blacks, and much more likely to have false negatives on whites, and the results are that blacks are more likely to treated more harshly by the system. But it's not biased because of the definition of bias I'm using!" Orwell would have loved "disparate impact isn't bias" :)

Clearly statistics terminology is confusing you. The definition of bias is E[\hat{\theta} - \theta]. The definition of disparate impact is a predictor computing different means/quantiles for different protected classes.

https://en.wikipedia.org/wiki/Bias_of_an_estimator

https://en.wikipedia.org/wiki/Disparate_impact

To understand this intuitively, here's a simple thought experiment.

Consider Captain Hindsight, a predictor which returns the right answer 100% of the time. By definition, E[\hat{theta} - \theta] = 0, i.e. zero bias. (Also zero variance.)

Now suppose that blacks have a higher recidivism rate (hardly implausible, ProPublica's analysis suggests they do with p Captain Hindsight - being 100% accurate and having no bias - must predict that blacks have a higher recidivism rate. Yet because Captain Hindsight predicts a higher recidivism rate for blacks, he now has disparate impact.

Seriously, you are calling standard mathematical terminology Orwellian? What's your angle here?

Re: Machine Bias

#35

Post by wyager »

I don't have an issue with using statistical analysis to direct crime prevention efforts. I think it's unconscionable to use statistical analysis for sentencing. We don't want Minority Report in real life.

Re: Machine Bias

#36

Post by zyxley »

yummyfajitas wrote: post #34
jdp23 wrote: post #29

Earlier quoted context omitted.

"Oh sure, this algorithm is much more likely to have false positives on blacks, and much more likely to have false negatives on whites, and the results are that blacks are more likely to treated more harshly by the system. But it's not biased because of the definition of bias I'm using!" Orwell would have loved "disparate impact isn't bias" :)

Clearly statistics terminology is confusing you. The definition of bias is E[\hat{\theta} - \theta]. The definition of disparate impact is a predictor computing different means/quantiles for different protected classes. https://en.wikipedia.org/wiki/Bias_of_an_estimator https://en.wikipedia.org/wiki/Disparate_impact To understand this intuitively, here's a simple thought experiment. Consider Captain Hindsight, a pred…

Your thought experiment here is incorrect, given that the analysis compares COMPAS results to actual recidivism rates and shows over- and under-prediction in comparison to them.

Re: Machine Bias

#37

Post by mdjeu »

kenjackson wrote: post #13
yummyfajitas wrote: post #9

Earlier quoted context omitted.

The validity of the algorithm can be - and apparently has been - reliably tested and been found to be useful and mostly unbiased. This analysis has been performed by both the algorithm's creators and highly adversarial third parties, such as the author of this article. Both found that whatever bias there is is small, and cannot be distinguished from random chance. For example, the author of this very article has done…

Lets be clear -- if the null hypothesis in this case is true (that there is no bias), and all other assumptions made are true, there is a slightly greater than 5.7% chance of obtaining this result (or something even more skewed). That's a great bar for publication of SCIENCE. It's not a great bar for hiding behind a proprietary algorithm used in sentencing. People talk about misuse of p-values, but this takes the cak…

This is in my professional area, and yummyfajitas is right on certain points. The reason these approaches started taking off at all is because the alternative, subjective decisions, don't generally work as well. There's plenty of meta-analyses to show this; that's why these risk systems get used.

Also, this analysis is certainly a useful addition to the literature on this system, but it's one analysis, and regardless of your philosophical stance on p-values, a p-value of .057 in the presence of multiple testing isn't the most convincing thing.

Having said that, the use of non-open predictive systems is a problem for criminal settings. Maybe this thing is biased, but the only way to find out and fix it is to do these sorts of analyses and have this sort of discussion.

The problem isn't the use of prediction systems, it's the use of them without open academic scrutiny, without correcting any biases that emerge.

Re: Machine Bias

#38

Post by pdkl95 »

yummyfajitas wrote: post #15
kenjackson wrote: post #14

Earlier quoted context omitted.

(From my above reply too, as it applies here also): Lets be clear -- if the null hypothesis in this case is true (that there is no bias), and all other assumptions made are true, there is a slightly greater than 5.7% chance of obtaining this result (or something even more skewed). That's a great bar for publication of SCIENCE. It's not a great bar for hiding behind a proprietary algorithm used in sentencing. People t…

If you want to criticize the details of her analysis, go ahead. I'm solidly in the Bayesian camp and I agree with you 100%. What I'd have done is computed posteriors on all these coefficients and then computed bayes factors/probability of bias. I'm confused though; the mood affiliation of your post somehow suggests that her less than perfect choice of a statistical methodology somehow supports her claims. Could you e…

> maximum likelihood

That may be grounds for a mistrial. Decisions about crimes are not judged by the "maximum likelihood".

> what specific analysis would convince you that this algorithm is predictive and non-biased

What is it going to take to convince you that the choice of model and which data to use as input is just as important as the analysis itself?

> race_factor

Depending on the situation, using race or other protected classes is illegal. One of the reasons we have a right to face our accusers is to provide an opportunity to challenge those accusations. Racial (or any other protected class) discrimination doesn't become legal when it is hidden behind an equation or algorithm. If the government wants to keep the method secret, then anything derived from those methods should be excluded.

> human biases

...are off topic. An algorithm needs to justify it's own existence.

> it's not very big

So you're fine with racial bias, as long as it only affects what you consider a "small" number of people.

> or perhaps black defends actually are more likely to commit crimes

/sigh/

Re: Machine Bias

#39

Post by yummyfajitas »

zyxley wrote: post #36
yummyfajitas wrote: post #34

Earlier quoted context omitted.

Clearly statistics terminology is confusing you. The definition of bias is E[\hat{\theta} - \theta]. The definition of disparate impact is a predictor computing different means/quantiles for different protected classes. https://en.wikipedia.org/wiki/Bias_of_an_estimator https://en.wikipedia.org/wiki/Disparate_impact To understand this intuitively, here's a simple thought experiment. Consider Captain Hindsight, a pred…

Your thought experiment here is incorrect, given that the analysis compares COMPAS results to actual recidivism rates and shows over- and under-prediction in comparison to them.

The thought experiment is a mathematical proof that the two concepts are causally unrelated, nothing more. I really suggest you brush up on your basic math - you seem to not be following along.

Your claims about the emirical means of recidivism rates do not prove what you think they prove. Different races might be misclassified at different rates for a variety of reasons - e.g., one race might be affected more by some high-variance predictor, or there could be composition effects (e.g. the pdf of blacks|high score might be different than whites|high score).

The way to factor out whether they scores are biased is to do the cox survival analysis with interaction terms. Which they did. You just don't like the result.

Could you clearly lay out the statistical argument that you believe implies that E[\hat{\theta} - \theta] > 0?

Re: Machine Bias

#40

Post by TheCoelacanth »

yummyfajitas wrote: post #33
TheCoelacanth wrote: post #28

Earlier quoted context omitted.

It is consistently giving incorrectly low scores to white subjects and consistently giving incorrectly high scores to black subjects. That is clearly bias, at least in the colloquial sense.

The degree to which it does this cannot be distinguished from random chance (p > 0.05). If the predictor were biased then you could build a more accurate score based on both the original scores and race_factorBlack:score_factorHigh (and other interaction terms). I.e. you'd be building a new bias in to cancel the old bias, leaving an accurate predictor. Their analysis doesn't show that this is possible.

p > 0.05 is the type of cutoff you would see to get published in a peer-reviewed paper. Such a high bar of evidence is not necessary in this situation. To prevail in a civil suit, a person harmed by this algorithm would only have to prove that is more likely than not that the algorithm is biased.
Post reply on HN