Earlier quoted context omitted.
> We can build algorithms to avoid racial bias, with awareness and effort. How confident are you we can train cops to do the same? How confident are you that we can build algorithms to do that? You can't just flatten the probability distribution based on race when many of the legitimate factors are correlated with it, any more than you could do so for gender or age or nationality. Unless you want an absurd false posi…
I believe the article clarifies that this system (much like its sibling, DNA database mining) is used to narrow the search space so that traditional (and comparatively more expensive) sleuthing can find the evidence needed for a conviction. The dragnet is, in a sense, more ruling people out than ruling people in.
Which is exactly the problem. The system spits out a list of 25 names, 20 of them have an alibi and only one of the remaining five had the capacity to commit the crime. So you've got your man, all you have to do is dig up some evidence. Unless the true perpetrator wasn't on the original list to begin with, or fabricated an alibi, in which case you're attempting to convict an innocent person. Which huge databases are ideal for doing because it's very easy to use selection bias to raise the apparent probability of random events.
It's like the reverse pyramid scheme. You choose a stock and send brochures to a million people telling half it will go up and half it will go down. Then it goes up or down and next week you send brochures to the half million people you sent the accurate prediction to last week, telling half of them that a different stock will go up and the other half that it will go down. By the end of ten weeks you have about a thousand people who think you can predict the market when all you've really done is to start with a large pool.
"99.9% accurate" in a city of two million means that 99.95% of the matches are false positives.
You can't start with a list of names and then look for evidence it was them. You have to start with the evidence and see who it points to.