Machine Bias
propublica.org
Machine Bias
1–10 of 61 posts
Re: Machine Bias
#2One of the most mind boggling sentences in that article was:
"On Sunday, Northpointe gave ProPublica the basics of its future-crime formula — which includes factors such as education levels, and whether a defendant has a job. It did not share the specific calculations, which it said are proprietary."
How on earth can you lock people up based on secret information? That is Kafka meets Minority Report.
Re: Machine Bias
#3This is totally fucked. Morally wrong, deeply unethical, and probably illegal – if you're adding punishment without having that additional punishment based on new evidence, isn't that like being treated guilty without proof? Obviously I'm not a lawyer, but how could anyone, let alone the whole huge set of people that led to these policies, think that applying group statistics to individuals to determine the severity of their punishment is ok?
On the other hand, these biaces (most notably the racial ones) exist in the process anyway, and now they're simply being codified and exposed. If these algorithms were published we could see exactly how much more punishment you get for being black in America versus being white.
Thanks again to ProPublica for an important piece of reporting; hopefully changes get made for the better.
Re: Machine Bias
#4According to propublicas own analysis, the claim of bias cannot be shown to be statistically significant. https://www.propublica.org/article/how-we-analyzed-the-compa...
This article is terrible data journalism and probably deliberately misleading.
Step 1: write down conclusion.
Step 2: do analysis.
Step 3: if analysis doesn't support conclusion, write down a bunch of anecdotes.
Really, here's her R script: https://github.com/propublica/compas-analysis/blob/master/Co...
Just read that. It's vastly better than this nonsensical article.
Re: Machine Bias
#5One of the most mind boggling sentences in that article was: "On Sunday, Northpointe gave ProPublica the basics of its future-crime formula — which includes factors such as education levels, and whether a defendant has a job. It did not share the specific calculations, which it said are proprietary." How on earth can you lock people up based on secret information? That is Kafka meets Minority Report.
This is done regularly. It's called "judicial discretion" - a judge uses a neural network so secret that even he doesn't understand it (in fact the entire scientific field of "neuroscience" exists to try and analyze it).
Variables used in the formula include details of the case, race/appearance of the defendant, and how recently lunch was at the time of sentencing. Unlike the ProPublica claims of racial bias (which are merely "almost statistically significant" at the p=0.05 level), the lunch bias is statistically significant at the p http://www.pnas.org/content/108/17/6889.full
This system sounds like a huge improvement.
Re: Machine Bias
#6Weapons of Math Destruction
http://boingboing.net/2016/01/06/weapons-of-math-destruction...
It's easy to hide agenda behind an algorithm; especially when the details of the algorithm are not publicly visible.
Re: Machine Bias
#7One of the most mind boggling sentences in that article was: "On Sunday, Northpointe gave ProPublica the basics of its future-crime formula — which includes factors such as education levels, and whether a defendant has a job. It did not share the specific calculations, which it said are proprietary." How on earth can you lock people up based on secret information? That is Kafka meets Minority Report.
This is done regularly. It's called "judicial discretion" - a judge uses a neural network so secret that even he doesn't understand it (in fact the entire scientific field of "neuroscience" exists to try and analyze it). Variables used in the formula include details of the case, race/appearance of the defendant, and how recently lunch was at the time of sentencing. Unlike the ProPublica claims of racial bias (which a…
As you note, the algorithm for judicial discretion is unknown. The algorithm for this software is fully known, just kept from the public.
Re: Machine Bias
#8This is totally fucked. Morally wrong, deeply unethical, and probably illegal – if you're adding punishment without having that additional punishment based on new evidence, isn't that like being treated guilty without proof? Obviously I'm not a lawyer, but how could anyone, let alone the whole huge set of people that led to these policies, think that applying group statistics to individuals to determine the severity…
That does sound like a good argument against it -- adding punishment without evidence... Could they argue that they're reducing sentences for those less likely to repeat? If they don't see "evidence" that the person will repeat then they give a reduced sentence (kindof like early parole). Still unethical crap because it pushes a race-based agenda (consciously or unconsciously). I'd say there's no difference, and would agreewith your argument. Also, not a lawyer. Technically they don't ask "are you black". They ask whether or not you had a parent incarcerated -- good for propagating a broken status quo. That question almost seems designed to "increase punishment without evidence". Regardless there shouldn't be any private algorithm deciding this and any public algorithm should be well scrutinized and validated for accuracy.
One thing is certain -- the federal government needs to shut these sentencing analysis companies down. At the very least heavy public audits. I'd say even libertarians would agree this is the definition of something that should be regulated.
Re: Machine Bias
#9Earlier quoted context omitted.
This is done regularly. It's called "judicial discretion" - a judge uses a neural network so secret that even he doesn't understand it (in fact the entire scientific field of "neuroscience" exists to try and analyze it). Variables used in the formula include details of the case, race/appearance of the defendant, and how recently lunch was at the time of sentencing. Unlike the ProPublica claims of racial bias (which a…
As you note, the algorithm for judicial discretion is unknown. The algorithm for this software is fully known, just kept from the public.
The validity of the algorithm can be - and apparently has been - reliably tested and been found to be useful and mostly unbiased. This analysis has been performed by both the algorithm's creators and highly adversarial third parties, such as the author of this article. Both found that whatever bias there is is small, and cannot be distinguished from random chance.
For example, the author of this very article has done such an analysis. Here's her R notebook:
https://github.com/propublica/compas-analysis/blob/master/Co...
Her analysis shows (within the limitations of the frequentist paradigm) that:
a) the predictor is useful - score_factorHigh and score_factorMedium both have p-values that are essentially zero.
b) The predictor is not racially biased that much - race_factorAfrican-American:score_factorHigh and the other bias terms have p-values that are > 0.05 .
Look, I'd love it if we required such algorithms to be open source. I'm a huge proponent of both open science and open government. Nevertheless, there is an entire discipline devoted to evaluating predictive algorithms without needing to care about their details - it's called "machine learning".
The wonderful thing about statistics is that even a highly biased person (such as the author of this article) can still reach a correct conclusion that goes against their biases.
Re: Machine Bias
#10According to propublicas own analysis, the claim of bias cannot be shown to be statistically significant. https://www.propublica.org/article/how-we-analyzed-the-compa... This article is terrible data journalism and probably deliberately misleading. Step 1: write down conclusion. Step 2: do analysis. Step 3: if analysis doesn't support conclusion, write down a bunch of anecdotes. Really, here's her R script: https://g…
They analyzed what they could -- the outcomes of the algorithm (recommendation) and the accuracy of those recommendations. They picked out specific examples, but the analysis was over the whole data set. I think you missed these relevant parts from the article:
> We obtained the risk scores assigned to more than 7,000 people arrested in Broward County, Florida, in 2013 and 2014 and checked to see how many were charged with new crimes over the next two years, the same benchmark used by the creators of the algorithm.
> The score proved remarkably unreliable in forecasting violent crime: Only 20 percent of the people predicted to commit violent crimes actually went on to do so.
> The formula was particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants. White defendants were mislabeled as low risk more often than black defendants.
> Could this disparity be explained by defendants’ prior crimes or the type of crimes they were arrested for? No. We ran a statistical test that isolated the effect of race from criminal history and recidivism, as well as from defendants’ age and gender.
> Black defendants were still 77 percent more likely to be pegged as at higher risk of committing a future violent crime and 45 percent more likely to be predicted to commit a future crime of any kind.