Make Algorithms Accountable

I think the problem is vastly more subtle and fundamental than the author of this piece (and ProPublica in general, given their previous coverage of the same story) is giving credit.

Software is in a dangerous place right now[1]. Half[2]-way between stupid and smart. Recent advances from fields like Machine Learning have moved the software out of world of the purely deterministic. Stochastic methods have given us software that can live in the gray that is reality; this has made it mighty. Particular subdomains that were once entirely the purview of human workers are rapidly moving towards automation. These algorithms have gotten good enough at what they do, accurate enough at what they do, that they are approaching the semantic-work-outsourcing limit that is 'trustworthiness'; You can treat them more like agents than tools, and trust that they do their job sufficiently well that you, the consumer of their work, need not worry about the details; you get to take the executive role of dealing only with the abstractions they provide. "Just tell me yes or no if we should do this". You implicitly trust that the system will "Do The Right Thing"; of course it will, it's got a fantastic resume with some great recommendations[3].

The problem is, the algs are also still dumb. Very dumb. They cannot model themselves. They cannot introspect. And, perhaps most crucially, they cannot interface with their newly-promoted executives in the lingua franca to explain why they're dumb. When you want to figure out why the intern decided that it would be a good idea to name all their variables some permutation of the words 'herp' and 'derp', you march over to their cube and ask them. A conversation occurs. Different perspectives are exchanged via a common protocol[4]. New knowledge is acquired. A mutual understanding is reached. When you want to figure out why your Facial Recognition Software isn't acknowledging Black People[5]... you go get a Masters in Statistics, Distributed System, Probability Theory, with a minor in Anthropology and Demographics. Then you spend a month reading code and running experiments. The software, briefly perceived as a trustable agent that knows how to do its job, suddenly becomes a tool again, because you can't just ask it why it screwed up so badly. And not just any tool; an incredibly complicated, fragile, and opaque tool, with a million different knobs and dials and a Gordian nest of pipework and conduits that would make even the bravest chaotician sweat a little bit. Even when you've open sourced the data and implementation and the deployment architecture and the napkins you've been scribbling hyperparameters on, the box is still pretty damn black.

And so, while auditability or accountability is important, it's only a small (and very, very, very hard) piece of the societal changes that might be needed. Changes in ethics (who goes to jail when a UAV confuses a hospital for a barracks? What does a smart gun do if its wielder pulls the trigger while pointing at a civilian?). Changes in focus (Is it correct to make such deterministic choices about the world? Maybe Hume was right and the predictability of human behavior is slightly harder than we currently state it to be[6]. Maybe accuracy is bounded lower than we like, and so justification should be the primary target). Changes in education (and a reduction in the magical thinking about computers. I hope some distant ancestors of mine finally see the day that computers are as boring and obvious as hammers).

I worry about focusing so much on accountability because as it stands now, a highly autidable system as defined by the article still needs deep domain knowledge to even begin theorizing about. All of these things above and more will probably need to shift as well, whether deliberately or not, in order to accommodate these smart-dumb tool-agent hybrids we have now, systems that we're building right now, that are just powerful enough to be dangerous.

---

[1] Every generation has said these exact words about every technological advance across every field for as long as we've been finding ways of rendering human beings obsolete. I'm sure there's some philologically reconstructible PIE for the phrase "those scientists are playing god". We've discovered it before, and I have no doubt we'll discover again solutions for these things, but that doesn't render the conversation in the interim any less meaningful.

[2] 'Half' is probably optimistic. Consider it an upper bound.

[3] Studied at MNIST... Tuned by Hinton and Ng... Deployed on three customer engagements with great KPIs... Very impressive stuff, Mr. Convolutional!

[4] I mean Natural Language here, but I suppose baseball bats and guttural screams may be other possible channels in this particular scenario.

[5] Or why your self-driving smart car crashed into a cement barrier

[6] I was trying to construct some pun about "the son not rising to the mistakes of the father tomorrow". It was pretty bad.

The example of redlining is actually a perfect rebuttal to "private individuals ought not to be accountable to anyone for their decisions."

Biases can be encoded in algorithms. For example, what if data they used to build their models was biased? It's conceivable that a bank's algorithm could be racist, and still be okay for their bottom line. Just letting the market decide is not enough.

Your argument in the second paragraph seems to be that arbitrary discrimination, for instance racial discrimination, is economically irrational; discriminatory companies will be out-competed by non-discriminatory companies.

But this ignores the historic reality of racial discrimination, which is that in some areas of the US, at some points in time, a significant number of white people were bigoted, and strongly (for no good reason) disliked black people. If you opened a restaurant that served people of all colors, your white customers would disappear. If your bank made loans to black people who wanted to buy homes in traditionally white neighborhoods, your white depositors would take their business elsewhere.

The civil rights laws of the 1960s made it economically feasible for businesses in these regions to stop discriminating; there were no longer any "white only" businesses for white bigots to patronize.

If you want to argue that in our more "enlightened" times, such anti-discrimination laws are no longer necessary, that's fine, but you will need to provide a certain amount of proof that things have changed.

Here is a paper [1] discussing the effects of the new European Union legislation mentioned in the article ("right to explanation"). Interesting related read, regardless of the validity of the numbers mentioned in this opinion piece.

[1] http://arxiv.org/pdf/1606.08813v2.pdf

I would love—assuming Congress had their heads in the right place—to make a small council (council to prevent Luddite or crazy Presidents/Congresses from completely wreaking havoc) that would understand tech stuff like we do. People who just comment "Yea that's not how this works," or "hey techies think this is a good idea." Because I give a lot of governments good credit on tech stuff (and especially local governments) but on the federal level I think we're a little lost and lobbyists get to influence what congress members believe.