Why are we using black box models in AI when we don’t need to? (2019)

I once worked at data science consultancy. They worked on a project that they ended up using random forest. But they also ran a simple single layer net to validate the results, scoring a similar accuracy (it was like 70%). They gave a presentation on a Friday to the company (it was a regular show and tell demo slot).

At the end, I asked them if they double checked whether the net weights corresponded to the random forest and almost everyone in the room just looked at me like I was an idiot.

I ended up causing a heated discussion in the pub later on because I’d actually done something similar for my masters thesis (it was for modelling XOR operations on binary inputs).

The company in question is notoriously short term focussed, so it shouldn’t really have been a surprise that they didn’t even try to think about it.

And that’s kind of the problem. Short term speculative projects don’t care about interpretability. A director at some clothing firm doesn’t care about the super nerdy maths, they like getting their numbers.

But, on the flip side, Explaining loss functions and backprop is also a no go for the directors. If you can’t help them understand an algorithm in 2 sentences then they often don’t buy it. So then explainability becomes a big thing (hence why random forest was implemented and not the net).

Sometimes, the best interpretable model is as good as a black box, and that's great.

When this is not the case, the trade-off is that one should see what's more important for the actual problem. Perhaps interpretability is not a big deal.

Another solution is to try to extract interpretability from the more accurate black box model with something like SHAP.

Very little of technology has anything to do with validating hypotheses.

> meaning that humans, even those who design them, cannot understand how variables are being combined to make predictions.

The intention is to not rely on the explanation to evaluate the effectiveness of the model. This does not preclude any of the infinite narratives that might explain the model.

This is fundamentally a cost saving mechanism to avoid hiring engineers to code heuristics useful to business. There is nothing related to science here at all. A "black box" model is fashionable to those who prefer to observe and not create meaning, even if the observed meaning is deeply flawed from a human perspective. After all, people spend money based on less all the time.

We can't "explain" how we recognize a particular person's face, for example.

Robust pattern recognition based on 100+'s of factors is just inherently black box.

Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction.

Black box doesn't necessarily mean bad. I think the relevant question is: how do we ensure machine learning is trained and verified in ways that don't encode bias, and only used with "intended" inputs, so a model isn't being trusted to make predictions on things it wasn't trained for?

And also: when do we want to decide to use alternatives like decision trees instead -- because despite being less accurate, they can have designers who can be held legally accountable and liable, which can be more important in certain situations?