Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Bias

propublica.org

Post reply on HN
Print view View original

51–60 of 61 posts

Re: Machine Bias

#51

Post by argonaut »

pessimizer wrote: post #48
mikeash wrote: post #25

Earlier quoted context omitted.

Punishment is always considered somewhat separately from the determination of guilt. The judge would already try to account for things like this when determining your sentence. They just do it in a deeply ad hoc and personal manner, where they just take a stab at it, try to account for things like how sorry you seem to be, apply guidelines, and come up with a number. This means that you might ultimately be punished f…

All this does is systematize those biases so that they can't be challenged like a judge with a record of bias can. The statistics that they choose to record create bias in and of themselves - by using race in the algorithm, you are building in the possibility that race influences criminality. If you built in favorite foods, some foods would end up resulting in higher sentences than others, just as if you built in pha…

But they can be challenged. That's why you're reading an article about it. If you have a judge that is biased, it is probably harder to challenge his sentences than if you had an algorithm that you proved was biased.

Re: Machine Bias

#52

Post by gleb »

thejefflarson wrote: post #24

Thanks for posting this. I encourage this crowd to to take a look at the methodology too: https://www.propublica.org/article/how-we-analyzed-the-compa...

Thanks for posting a link to the methodology.

Does this sentence "Northpointe does offer a custom test for women, but it is not in use in Broward County. " imply that the base COMPAS model does not take gender into account?

Re: Machine Bias

#53

Post by [deleted] »

yummyfajitas wrote: post #4

According to propublicas own analysis, the claim of bias cannot be shown to be statistically significant. https://www.propublica.org/article/how-we-analyzed-the-compa... This article is terrible data journalism and probably deliberately misleading. Step 1: write down conclusion. Step 2: do analysis. Step 3: if analysis doesn't support conclusion, write down a bunch of anecdotes. Really, here's her R script: https://g…

[deleted]

Re: Machine Bias

#54

Post by selectron »

TheCoelacanth wrote: post #28
yummyfajitas wrote: post #20

Earlier quoted context omitted.

None of these things are evidence of bias. The algorithm is biased if it's giving the wrong score due to race or redundantly encoded race. To show that the algorithm is biased, you need to show that (score, race) pairs are more predictive than (score, ) singletons. Line [36] and [46] both attempt to address this question. The only one of these which is statistically significant is "race_factorOther:score_factorHigh"…

It is consistently giving incorrectly low scores to white subjects and consistently giving incorrectly high scores to black subjects. That is clearly bias, at least in the colloquial sense.

The real question though is whether this is because the model is biased, or whether it is because the two populations have different levels of risk. If group A had a higher risk than group B, than I would expect the model to have a higher rate of false negatives for group A than for group B. This is just because the model is more likely to (correctly) assign members of group A as high risk, and some of these classifications will be wrong. To check for bias you have to control for this base rate difference.

Re: Machine Bias

#55

Post by gleb »

thejefflarson wrote: post #24

Thanks for posting this. I encourage this crowd to to take a look at the methodology too: https://www.propublica.org/article/how-we-analyzed-the-compa...

Are you sure what you found is not just Simpson's paradox?

When I look at the 2 KM plots for white/blacks, they are mostly the same. It's pretty clear that the model is not prejudiced against blacks, in fact it's somewhat prejudiced against whites. [1]

Your main editorial claim is that whites tend to be misclassified as "good" and blacks as "bad."

But I think what's actually happening is that algorithm is more likely to misclassify low_risk as "good", and high_risk as "bad".[2] Combine that with vastly more whites than blacks being low_risk (as you show earlier) and you get the observed "injustice".

I'll also note that the KM for whites flatten out at 2 years, unlike for blacks. This is actually a big deal if statistically significant. But that's a separate conversation.

Footnotes:

1 - this is acknowledged in methodology page "black defendants who scored higher did recidivate slightly more often than white defendants (63 percent vs. 59 percent)."

2 - why that is I don't yet fully understand (and I'd like to) but it looks's to be simple math that follows from low risk mostly not recidivating, and high risk mostly yes recidivating

Re: Machine Bias

#56

Post by daveguy »

yummyfajitas wrote: post #11
daveguy wrote: post #10

Earlier quoted context omitted.

They analyzed what they could -- the outcomes of the algorithm (recommendation) and the accuracy of those recommendations. They picked out specific examples, but the analysis was over the whole data set. I think you missed these relevant parts from the article: > We obtained the risk scores assigned to more than 7,000 people arrested in Broward County, Florida, in 2013 and 2014 and checked to see how many were charge…

Go read the description of the statistical analysis or just view their R notebook: https://github.com/propublica/compas-analysis/blob/master/Co... Their own analysis shows that (p ~= 0) that high and medium risk factors are predictive. They also showed that the racial bias terms (race_factorAfrican-American:score_factorHigh, etc) are probably not predictive (p > 0.05). Your quotes are not evidence of bias, though I s…

I read through the entire analysis. It appears that you stopped reading after you saw a p-value that supported your bias. That is bias in the sense of pre-conceived notion. You then proceeded to pedantically argue that the well demonstrated bias of the algorithm (more false positives for blacks than whites about 40% vs 20%) does not exist because of a p-value that came in between 0.05 to 0.1 instead of below 0.05.

Please let me know when your reading comprehension catches up with your mediocre statistics comprehension.

Maybe you just didn't realize that the 20-20 hindsight data -- prediction vs recidivism -- is included right there in the analysis. Or maybe you did realize it later and just decided you'd dug in so much that you didn't want to admit your ignorance.

Or maybe you still haven't comprehended the difference between the meanings of the word bias.

Re: Machine Bias

#57

Post by Malarkey73 »

cwilkes wrote: post #21
Malarkey73 wrote: post #2

One of the most mind boggling sentences in that article was: "On Sunday, Northpointe gave ProPublica the basics of its future-crime formula — which includes factors such as education levels, and whether a defendant has a job. It did not share the specific calculations, which it said are proprietary." How on earth can you lock people up based on secret information? That is Kafka meets Minority Report.

What if it came out of a neural net or some other system that can't be easily explained? There's no real "specific calculation" to show. Now if they were using decision trees, i.e. If the person has 3 or more felonies they get a 5 rating, that could be presented. I'm curious about how much of a feedback loop this process has. The model was probably trained on old data and never updated. Also how does it take into acc…

I have this very concern about using SVM in medical research.

I also worry greatly about diagnostic predictive models that maximise overall prediction success but don't balance the relative consequences of false positives and false negatives.

Re: Machine Bias

#59

Post by kenjackson »

mdjeu wrote: post #37
kenjackson wrote: post #13

Earlier quoted context omitted.

Lets be clear -- if the null hypothesis in this case is true (that there is no bias), and all other assumptions made are true, there is a slightly greater than 5.7% chance of obtaining this result (or something even more skewed). That's a great bar for publication of SCIENCE. It's not a great bar for hiding behind a proprietary algorithm used in sentencing. People talk about misuse of p-values, but this takes the cak…

This is in my professional area, and yummyfajitas is right on certain points. The reason these approaches started taking off at all is because the alternative, subjective decisions, don't generally work as well. There's plenty of meta-analyses to show this; that's why these risk systems get used. Also, this analysis is certainly a useful addition to the literature on this system, but it's one analysis, and regardless…

but it's one analysis, and regardless of your philosophical stance on p-values, a p-value of .057 in the presence of multiple testing isn't the most convincing thing.

I agree in general. But when you have one data point and it relates to bias in a system a p-value of .057, suggesting there is bias is more compelling than the null hypothesis. Especially when other independent a-priori evidence seems to also point against the null hypothesis.

Re: Machine Bias

#60

Post by michaelbuddy »

yummyfajitas wrote: post #46
zyxley wrote: post #45

Earlier quoted context omitted.

> Could you clearly lay out the statistical argument that you believe implies that E[\hat{\theta} - \theta] > 0? Again: I don't care about your domain-specific definition of "bias". I care about whether the end result of this secret algorithm is unequal and inaccurate treatment of different demographic groups.

inaccurate treatment of different demographic groups. This is exactly what the standard mathematical definition of bias (restricted to a given group) addresses. The authors of this article ran exactly that analysis - see lines [36] and [46]. I know that you are trying to retreat from statistics, since the stats don't support your mood affiliation, but don't retreat to "accuracy". Retreat to something vague and undefi…

> since the stats don't support your mood affiliation

This is the most sophisticated and high powered zing I've heard in a while. Kudos.

Post reply on HN