Every decision made by an algorithm should make its inputs clear, its criteria for interpreting those inputs clear, and its judgement to be disputable. If you can't get your black-box neural network to do so, then perhaps it shouldn't be making life-changing decisions for other people.
There's a startup that's doing sentiment analysis of social media posts, to measure how 'risky' a babysitter is - ad likely does so in a biased manner. [1]
It's illegal to use such a system, for purposes of vetting an employee, yet their entire business model revolves around families using them to vet babysitters.
[1] https://gizmodo.com/predictim-claims-its-ai-can-flag-risky-b...