Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Bias

propublica.org

Post reply on HN
Print view View original

21–30 of 61 posts

Re: Machine Bias

#21

Post by cwilkes »

Malarkey73 wrote: post #2

One of the most mind boggling sentences in that article was: "On Sunday, Northpointe gave ProPublica the basics of its future-crime formula — which includes factors such as education levels, and whether a defendant has a job. It did not share the specific calculations, which it said are proprietary." How on earth can you lock people up based on secret information? That is Kafka meets Minority Report.

What if it came out of a neural net or some other system that can't be easily explained? There's no real "specific calculation" to show.

Now if they were using decision trees, i.e. If the person has 3 or more felonies they get a 5 rating, that could be presented.

I'm curious about how much of a feedback loop this process has. The model was probably trained on old data and never updated. Also how does it take into account features that it doesn't know about (the article mentions one guy turning to Christianity)? I doubt if there is a mechanism for people to be asked why they did or did not reoffend. Even if they did how much should it be trusted?

Re: Machine Bias

#22

Post by mattkrause »

yummyfajitas wrote: post #5
Malarkey73 wrote: post #2

One of the most mind boggling sentences in that article was: "On Sunday, Northpointe gave ProPublica the basics of its future-crime formula — which includes factors such as education levels, and whether a defendant has a job. It did not share the specific calculations, which it said are proprietary." How on earth can you lock people up based on secret information? That is Kafka meets Minority Report.

This is done regularly. It's called "judicial discretion" - a judge uses a neural network so secret that even he doesn't understand it (in fact the entire scientific field of "neuroscience" exists to try and analyze it). Variables used in the formula include details of the case, race/appearance of the defendant, and how recently lunch was at the time of sentencing. Unlike the ProPublica claims of racial bias (which a…

Just FYI: The lunch paper has very serious problems as described in this reply, also published in PNAS: http://www.pnas.org/content/108/42/E833.full)

In particular, the cases are heard in a particular order. For each prison, the prisoners with counsel go before those who are representing themselves. As in the US, those representing themselves typically fair worse. The judges try to finish an entire prison's worth of hearings before a meal, so the least-likely-to-succeed cases are typically assigned to spots right before a break.

There are some other bits of weirdness in the original data too. They found a statistically significant association between the ordinal position (e.g., 1st, 2nd, ..., last) and the parole board's decision, but failed to find any effect of actual time elapsed (e.g., in minutes), even though the latter is much more compatible with a physiological hypothesis like running out of glucose.

Re: Machine Bias

#23

Post by yummyfajitas »

mattkrause wrote: post #22
yummyfajitas wrote: post #5

Earlier quoted context omitted.

This is done regularly. It's called "judicial discretion" - a judge uses a neural network so secret that even he doesn't understand it (in fact the entire scientific field of "neuroscience" exists to try and analyze it). Variables used in the formula include details of the case, race/appearance of the defendant, and how recently lunch was at the time of sentencing. Unlike the ProPublica claims of racial bias (which a…

Just FYI: The lunch paper has very serious problems as described in this reply, also published in PNAS: http://www.pnas.org/content/108/42/E833.full ) In particular, the cases are heard in a particular order. For each prison, the prisoners with counsel go before those who are representing themselves. As in the US, those representing themselves typically fair worse. The judges try to finish an entire prison's worth of…

Interesting, I was unaware. I need to associate more uncertainty to my beliefs about how terrible humans are at making decisions.

Re: Machine Bias

#25

Post by mikeash »

kough wrote: post #3

This is totally fucked. Morally wrong, deeply unethical, and probably illegal – if you're adding punishment without having that additional punishment based on new evidence, isn't that like being treated guilty without proof? Obviously I'm not a lawyer, but how could anyone, let alone the whole huge set of people that led to these policies, think that applying group statistics to individuals to determine the severity…

Punishment is always considered somewhat separately from the determination of guilt. The judge would already try to account for things like this when determining your sentence. They just do it in a deeply ad hoc and personal manner, where they just take a stab at it, try to account for things like how sorry you seem to be, apply guidelines, and come up with a number. This means that you might ultimately be punished for the judge not having a good breakfast:

http://www.scientificamerican.com/article/lunchtime-leniency...

And of course it goes without saying that judges will be affected by their biases, racial and otherwise.

I'm not sure what to do about it, though. Handing down the exact same punishment for every single person who commits a particular crime seems too blind. But any variation is going to be problematic.

Re: Machine Bias

#26

Post by SixSigma »

cwilkes wrote: post #21
Malarkey73 wrote: post #2

One of the most mind boggling sentences in that article was: "On Sunday, Northpointe gave ProPublica the basics of its future-crime formula — which includes factors such as education levels, and whether a defendant has a job. It did not share the specific calculations, which it said are proprietary." How on earth can you lock people up based on secret information? That is Kafka meets Minority Report.

What if it came out of a neural net or some other system that can't be easily explained? There's no real "specific calculation" to show. Now if they were using decision trees, i.e. If the person has 3 or more felonies they get a 5 rating, that could be presented. I'm curious about how much of a feedback loop this process has. The model was probably trained on old data and never updated. Also how does it take into acc…

Neural nets may be opaque but they are not secret.

Re: Machine Bias

#27

Post by zyxley »

yummyfajitas wrote: post #20
zyxley wrote: post #18

Earlier quoted context omitted.

Uh... it's all right there in your link, across several sections that analyze specific parts of the data. > Black defendants are 45% more likely than white defendants to receive a higher score correcting for the seriousness of their crime, previous arrests, and future criminal behavior. > Women are 19.4% more likely than men to get a higher score. > Most surprisingly, people under 25 are 2.5 times as likely to get a…

None of these things are evidence of bias. The algorithm is biased if it's giving the wrong score due to race or redundantly encoded race. To show that the algorithm is biased, you need to show that (score, race) pairs are more predictive than (score, ) singletons. Line [36] and [46] both attempt to address this question. The only one of these which is statistically significant is "race_factorOther:score_factorHigh"…

> but do not show bias. At best they show disparate impact

I have no interest in playing but-what-does-the-exact-dictionary-definition-say semantics games.

Re: Machine Bias

#28

Post by TheCoelacanth »

yummyfajitas wrote: post #20
zyxley wrote: post #18

Earlier quoted context omitted.

Uh... it's all right there in your link, across several sections that analyze specific parts of the data. > Black defendants are 45% more likely than white defendants to receive a higher score correcting for the seriousness of their crime, previous arrests, and future criminal behavior. > Women are 19.4% more likely than men to get a higher score. > Most surprisingly, people under 25 are 2.5 times as likely to get a…

None of these things are evidence of bias. The algorithm is biased if it's giving the wrong score due to race or redundantly encoded race. To show that the algorithm is biased, you need to show that (score, race) pairs are more predictive than (score, ) singletons. Line [36] and [46] both attempt to address this question. The only one of these which is statistically significant is "race_factorOther:score_factorHigh"…

It is consistently giving incorrectly low scores to white subjects and consistently giving incorrectly high scores to black subjects. That is clearly bias, at least in the colloquial sense.

Re: Machine Bias

#29

Post by jdp23 »

yummyfajitas wrote: post #20
zyxley wrote: post #18

Earlier quoted context omitted.

Uh... it's all right there in your link, across several sections that analyze specific parts of the data. > Black defendants are 45% more likely than white defendants to receive a higher score correcting for the seriousness of their crime, previous arrests, and future criminal behavior. > Women are 19.4% more likely than men to get a higher score. > Most surprisingly, people under 25 are 2.5 times as likely to get a…

None of these things are evidence of bias. The algorithm is biased if it's giving the wrong score due to race or redundantly encoded race. To show that the algorithm is biased, you need to show that (score, race) pairs are more predictive than (score, ) singletons. Line [36] and [46] both attempt to address this question. The only one of these which is statistically significant is "race_factorOther:score_factorHigh"…

"Oh sure, this algorithm is much more likely to have false positives on blacks, and much more likely to have false negatives on whites, and the results are that blacks are more likely to treated more harshly by the system. But it's not biased because of the definition of bias I'm using!"

Orwell would have loved "disparate impact isn't bias" :)

Re: Machine Bias

#30

Post by jdp23 »

zyxley wrote: post #27
yummyfajitas wrote: post #20

Earlier quoted context omitted.

None of these things are evidence of bias. The algorithm is biased if it's giving the wrong score due to race or redundantly encoded race. To show that the algorithm is biased, you need to show that (score, race) pairs are more predictive than (score, ) singletons. Line [36] and [46] both attempt to address this question. The only one of these which is statistically significant is "race_factorOther:score_factorHigh"…

> but do not show bias. At best they show disparate impact I have no interest in playing but-what-does-the-exact-dictionary-definition-say semantics games.

The funny thing is that the dictionary definition supports your point, not yummyfajitas.

bias: prejudice in favor of or against one thing, person, or group compared with another, usually in a way considered to be unfair.

Post reply on HN