In statistics there is a thing called bias, which can cause a lot of problems if not correctly handled.
An example of bias is historically most black people default on their loans. ML is deployed to predict if someone might default on a loan. Because ML does not understand bias, it sees the person is black and denies them for that, purely off of the fact that black people historically have defaulted more on their loans.
Bias is when ML sees something not relevant as a pattern and uses it as a feature to determine the future. Instead if race was filtered out, it might have seen historically most black people who got a loan were weak in other areas, like income or income stability or something else that actually factors in. It then could predict the future with a higher level of accuracy.
Police bias is worse than other industries, because it creates a feedback loop. If you think a black person is more likely to commit a crime, and you put more resources into that, then you're going to find more crime. This increases bias and it feeds on itself.
It seems the common fear on YC is the algorithms in predictive policing have a strong bias, causing problems. This is a legitimate risk, but imho not because of the algorithms but because of how they're used. They blindly give insights and police officers use this to increase bias, amplifying the issues we currently have.
On the NSA level the algorithms, which are not predictive policing, deal with bias much better and work quite well. They're scary good, better than having someone watching you at all times. Though, I guess that's a bit off topic.