Courts are using risk-assessment software to sentence criminals • Hacker News Board

Re: Courts are using risk-assessment software to sentence criminals

#51

Post by smnrchrds » Tue, Apr 18, 2017, 5:34 PM UTC

mcherm wrote: post #29

One of my big concerns is whether the algorithms are institutionalizing racism, with legal decisions that make it impossible to challenge these. After all, the algorithm has been trained on information about recidivism that was collected in a world where racism skews arrest rates, conviction rates, and sentencing.[1] That means the algorithm is almost certainly baking in a racial bias. Now, I'm sure they aren't fooli…

You are right to be worried.

"It found that widely used software that assessed the risk of recidivism in criminals was twice as likely to mistakenly flag black defendants as being at a higher risk of committing future crimes. It was also twice as likely to incorrectly flag white defendants as low risk."

Source: https://www.nytimes.com/2016/06/26/opinion/sunday/artificial...

Re: Courts are using risk-assessment software to sentence criminals

#52

Post by humanrebar » Tue, Apr 18, 2017, 5:35 PM UTC

mcherm wrote: post #29

One of my big concerns is whether the algorithms are institutionalizing racism, with legal decisions that make it impossible to challenge these. After all, the algorithm has been trained on information about recidivism that was collected in a world where racism skews arrest rates, conviction rates, and sentencing.[1] That means the algorithm is almost certainly baking in a racial bias. Now, I'm sure they aren't fooli…

> ...other correlated factors will allow the algorithm to continue to enforce this racism, but now with legal immunity.

I'm sympathetic, but how do we keep from causing more harm than good here? It seems like blindly ignoring all data that correlates with race is also bad. We do this in some circumstances already (i.e., IQ tests when doing hiring or admissions) and I think it's a defensible point that we're worse off for it.

Re: Courts are using risk-assessment software to sentence criminals

#53

Post by humanrebar » Tue, Apr 18, 2017, 5:37 PM UTC

dsfyu404ed wrote: post #19

bmh_ca wrote: post #9

I wonder if the algorithm is evidence-based, and learns from the results of prior decisions. In which case it is possible it would eventually discover that in the USA incarceration is very strongly linked to recidivism. It follows that the algorithm might refuse to incarcerate many convicts. Which is arguably exactly what the algorithm should do, namely what politicians will/can not: employ evidence to advance the me…

The problem with that is that criminality has it's own positive feedback loop. More crime in an area -> more cops -> more convictions for little stuff -> more big sentences handed out -> more lives ruined -> more crime -> goto start.

> more convictions for little stuff

Seems like an easier way to break that loop is to decriminalize the "little stuff" instead of capriciously enforcing it. No algorithm is going to be able to do that.

Re: Courts are using risk-assessment software to sentence criminals

#54

Post by pooya72 » Tue, Apr 18, 2017, 5:39 PM UTC

mcherm wrote: post #29

One of my big concerns is whether the algorithms are institutionalizing racism, with legal decisions that make it impossible to challenge these. After all, the algorithm has been trained on information about recidivism that was collected in a world where racism skews arrest rates, conviction rates, and sentencing.[1] That means the algorithm is almost certainly baking in a racial bias. Now, I'm sure they aren't fooli…

Yeah, there's a good book on this very issue called "Weapons of Math Destruction."[1] The author did an interview on a podcast and discusses how algorithms used by these judges reflect our racial biases.[2] One example that I remember is that the algorithm takes into consideration one's zip code, which correlates strongly with race in the US.

[1]: https://weaponsofmathdestructionbook.com/ [2]: http://www.econtalk.org/archives/2016/10/cathy_oneil_on_1.ht...

Re: Courts are using risk-assessment software to sentence criminals

#55

Post by cauterized » Tue, Apr 18, 2017, 5:47 PM UTC

mcherm wrote: post #29

One of my big concerns is whether the algorithms are institutionalizing racism, with legal decisions that make it impossible to challenge these. After all, the algorithm has been trained on information about recidivism that was collected in a world where racism skews arrest rates, conviction rates, and sentencing.[1] That means the algorithm is almost certainly baking in a racial bias. Now, I'm sure they aren't fooli…

Not to mention that discrimination in hiring decreases opportunities for minorities. Which in turn increases recidivism.

Re: Courts are using risk-assessment software to sentence criminals

#56

Post by hackuser » Tue, Apr 18, 2017, 5:51 PM UTC

humanrebar wrote: post #52

mcherm wrote: post #29

One of my big concerns is whether the algorithms are institutionalizing racism, with legal decisions that make it impossible to challenge these. After all, the algorithm has been trained on information about recidivism that was collected in a world where racism skews arrest rates, conviction rates, and sentencing.[1] That means the algorithm is almost certainly baking in a racial bias. Now, I'm sure they aren't fooli…

> ...other correlated factors will allow the algorithm to continue to enforce this racism, but now with legal immunity. I'm sympathetic, but how do we keep from causing more harm than good here? It seems like blindly ignoring all data that correlates with race is also bad. We do this in some circumstances already (i.e., IQ tests when doing hiring or admissions) and I think it's a defensible point that we're worse off…

> blindly ignoring all data that correlates with race is also bad

This is a very theoretical, futuristic red herring. Race factors into decision-making far, far too much. It's like worrying that if someone developed a perfect, bug-free crypto library, then everyone would use it and we'd have a monoculture - not really a present concern, and in fact a good problem to have (nor is it an eloquent analogy, but I have to run).

Re: Courts are using risk-assessment software to sentence criminals

#57

Post by hackuser » Tue, Apr 18, 2017, 5:54 PM UTC

rayiner wrote: post #8

> These algorithmic outputs inform decisions about bail, sentencing, and parole. Each tool aspires to improve on the accuracy of human decision-making that allows for a better allocation of finite resources. It's really not clear to me that much is gained from having very precise decisions made about bail and sentencing. Trying to predict the future is a fool's errand, whether a judge does it or a computer. It'd be b…

> It'd be better to just set fair, uniform standards

Aren't uniform standards as likely as a universal algorithm? There always will be messy edge cases where human judgment is needed. Mandatory sentencing rules worked out badly (though the current U.S. Attorney General Jeff Sessions is still in favor of them!)

Or am I misunderstanding you?

Re: Courts are using risk-assessment software to sentence criminals

#58

Post by hackuser » Tue, Apr 18, 2017, 5:54 PM UTC

rayiner wrote: post #8

> These algorithmic outputs inform decisions about bail, sentencing, and parole. Each tool aspires to improve on the accuracy of human decision-making that allows for a better allocation of finite resources. It's really not clear to me that much is gained from having very precise decisions made about bail and sentencing. Trying to predict the future is a fool's errand, whether a judge does it or a computer. It'd be b…

> "Scientism"

"Numerology" might be a better word.

Re: Courts are using risk-assessment software to sentence criminals

#59

Post by openasocket » Tue, Apr 18, 2017, 6:17 PM UTC

Using traditional machine learning techniques for this purpose is a non-starter and completely unacceptable. Neural networks are just a black box, and don't produce an inspectable justification or reasoning. The best you can say is that the model correctly predicts recidivism in X% of cases for some sample.

I'm not opposed to using other algorithmic methods, but the algorithm needs to be transparent. Though that would be difficult to do outside of some pretty tightly controlled parameters. We can't currently make a system that can take into account arbitrary facts about the case and weigh ethical implications.

Re: Courts are using risk-assessment software to sentence criminals

#60

Post by openasocket » Tue, Apr 18, 2017, 6:20 PM UTC

pizzetta wrote: post #3

Instead of asking to stop its use, why not ask that it be "supervised" up to the point where its results beat the average judge in a given area of law?

What does supervision mean in this case? To actually gather data, you'd have to give this system power to actually sentence people. And sentencing people to jail based on the decisions of an explicitly work-in-progress AI purely for the purposes of experimentation is a serious ethical violation.