Interpretability is probably one of the most misunderstood topics in AI. Interpretability is fundamentally not a math or statistics challenge, it's a Human Computer Interaction (HCI) problem and needs to be studied in a similar manner, with user studies. The fundemental goal of interpretability is to help users achieve some concrete goal such as detecting bias, identifying bad models, and debugging. This is similar how we can better design keyboards to help users achieve faster typing speed. Unfortunately, most interpretability studies don't actually bother testing if their methods actually work and I am highly skeptical that things like "white box models" or salience maps or attention weights or whatever latest methods can actually achieve the aims that interpretability proponents claim they do. I would love to see a simple randomized user study that would emperically show a concrete improvement in something like bias detection or error detection.
Why are we using black box models in AI when we don’t need to? (2019)
121–130 of 200 posts
Re: Why are we using black box models in AI when we don’t need to? (2019)
#122Earlier quoted context omitted.
> I think it's important to note that human pattern recognition is basically black-box as well. Agreed. But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.) So maybe we don't necessarily need fully interpretable models but simply a way to…
> even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.) You punctuate the second sentence as though it were of secondary importance. But in many cases, we have little ability to figure out how we came to a conclusion, while being much better at fabricating…
> But in many cases, we have little ability to figure out how we came to a conclusion, while being much better at fabricating plausible and politically acceptable answers.
Hmm, too me it feels like I can explain the reasons why I came to a conclusion in many (but certainly not all) cases. You "just" need to clearly identify your feelings and emotions and separate them from your rational arguments.
Anyway, these are our own shortcomings and of course don't have to be adopted by any artificially built black-box model.
Re: Why are we using black box models in AI when we don’t need to? (2019)
#123Earlier quoted context omitted.
"The point is that the explanation given does not correspond to the process through which the decision was truly reached." Sure, but that doesn't mean "a weighted combination of feelings" is an accurate description of the true process. That's tendentious and speculative. What I think we can be sure of is that there are parts of our decision making that we are not consciously aware of, but precisely because of that, o…
Parent wasn't saying that the weighting process was rational. The weights are also feelings.
As long as we don't know what the weighting process is, we shouldn't make unfounded statements about how it works. You're asserting that it exists, but nobody knows anything about what it is. It's a program that you haven't read the source code for.
Re: Why are we using black box models in AI when we don’t need to? (2019)
#124Earlier quoted context omitted.
But that explanation is likely full of cognitive bias, and all sorts of subjective experience. Epistemology and phenomenology are philosophical fields that deal with this. By no means is it a solved problem.
Object recognition is firmly in the realm of Neuroscience. We've had a good idea what the eye-V1 path is doing in mammals for a few decades now. To the point where we are implanting chips into the brain to (crudely) re-establish vision in blind patients. Though the research is very much on-going, it is being better understood with each passing year.
Respectfully, unless I'm not understanding what you mean by this, that seems very far from object recognition. You're replicating the data stream that would normally be coming from the optic nerve/retina, but that's not the same thing as understanding how the data of an object's image relates recognition of that object. If I've never seen the entire Mona Lisa before (but maybe have seen portions of it), your proposed system could force me to see the painting but it can't force me to recognize it as something I have seen, nor force me to know that it is a famous painting by Leonardo da Vinci currently on display at the Louvre.
Re: Why are we using black box models in AI when we don’t need to? (2019)
#125I am worried about the recent trend of "ethical AI", "interpretable models" etc. IMO it attracts people that can't come with SOTA advances in real problems and its their "easier, vague target" to hit and finish their PhDs while getting published in top journals. Those same people will likely at some point call for a strict regulation of AI using their underwhelming models to keep their advantage, faking results of th…
> IMO it attracts people that can't come with SOTA advances in real problems and its their "easier, vague target" to hit and finish their PhDs while getting published in top journals.
I'm also pretty wary of interpretability/explainability research in AI. Work on robustness and safety tends to be a bit better (those communities at least mathematically characterize their goals and contributions, and propose reasonable benchmarks).
But I'm also skeptical of a lot of modern deep learning research in general.
In particular, your critique goes both directions.
If I had a penny for every dissertation in the past few years that boiled down to "I built an absurdly over-fit/wrongly-fit model in domain D and claimed it beats SoTA in that domain. Unfortunately, I never took a course about D and ignored or wildly misused that domain's competitions/benchmarks. No one in that community took my amazing work seriously, so I submitted to NeurIPS/AAAI/ICML/IJCAI/... instead. On the Nth resubmission I got some reviewers who don't know anything about D but lose their minds over anything with the word deep (conv, residual, variational, adversarial, ... depending on the year) in the title. So, now I have a PhD in 'AI for D' but everyone doing research in D rolls their eyes at my work."
> Those same people will likely at some point call for a strict regulation of AI...
The most effectual calls for regulation of the software industry will not come from technologists. The call will come from politicians in the vein of, e.g., Josh Hawley or Elizabeth Warren. Those politicians have very specific goals and motivations which do not align with those of researchers doing interpretability/explainability research. If the tech industry is regulated, it's extremely unlikely that those regulations will be based upon proposals from STEM PhDs. At least in the USA.
> faking results of their interpretable models
Jumping from "this work is probably not valuable" to "this entire research community are a bunch of fraudsters" is a pretty big jump. Do you have any evidence of this happening?
Re: Why are we using black box models in AI when we don’t need to? (2019)
#126I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…
Sure, human pattern recognition is poorly understood, but it's a system that evolved over millions of years. We should expect more from something we ourselves designed.
The question of why we would choose a black box model over something interpretable that works equally or nearly as well still needs to be asked, especially if there is a misconception of black box models working better than interpretable models due to current trends and the hype cycle.
> held legally accountable
I think this misses the point. It is not about accountability but about getting the right results. The right question is where we need an interpretable model to eliminate biases or otherwise satisfy ethical requirements. Often you can't know whether a decision was ethical without knowing how it was reached. Would you accept a trial verdict given by AI? What about a lending decision, or a job promotion? If you need to justify a decision and not merely reach a decision, you need something interpretable.
Re: Why are we using black box models in AI when we don’t need to? (2019)
#127I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…
I think it's also important to remember a lot of human models were abandoned because of the inherent bias and black-boxed nature of them (feelings, subjectivity, etc.).
This is one of the reasons the scientific method blossomed: objectivity, rigor, transparency, reproducibility, etc. Black boxes can lead to bad decisions because it's difficult to question the process leading to decisions or highlight flaws in conclusions. When a model is developed with more rigor, it can be openly critiqued.
Instead, we have models running across such massive datasets with so many degrees of freedom that we have no feasible way of isolating problems when we see or suspect certain conclusions are amiss. Instead, we throw more data at it or train the model around those edge cases.
To be clear, I'm not saying ANN/DNN models are bad, just that we need to understand what we're getting into when we use them and recognize effects that unknown error bounds may cause.
If, when the model fails to correctly classify a new data point, the result is your photo editing tool can't remove a background person properly... then so be it, no harm no foul. If the result is that the algorithm classified a face incorrectly with high certainty and lands someone in prison as a result (we're not there, yet) then we need to understand our method has potential unknown flaws and we should proceed carefully before jumping to conclusions.
Re: Why are we using black box models in AI when we don’t need to? (2019)
#128I think it's important to note that human pattern recognition is basically black-box as well. We can't "explain" how we recognize a particular person's face, for example. Robust pattern recognition based on 100+'s of factors is just inherently black box. Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization a…
Yes, that’s true. But human decisions have accountability - algorithms don’t have accountability. If entities who deploy algorithms would be legally required to take accountability for algorithmic decisions, there would be no issue with black box algorithms.
Re: Why are we using black box models in AI when we don’t need to? (2019)
#129The main advantage of a blackbox ML solution is shorter development time to a useful performance level. Creating a transparent, explainable solution typically takes more time, more work, and a higher level of expertise to get to the same performance level. If the problem is complicated and the cost of a mistake is low, then your best approach today is likely to be blackbox. If the cost of a mistake is high, you shoul…
> The main advantage of a blackbox ML solution is shorter development time to a useful performance level. I think that's kinda true but kinda false. Its true that deep learning often makes feature engineering moot. however, a lot of deep learning projects takes machine learning engineers and applied scientists along with a host of other support engineers + hardware costs ( I seen people at work say: I only used a 16…
> Its true that deep learning often makes feature engineering moot. however, a lot of deep learning projects takes machine learning engineers and applied scientists along with a host of other support engineers + hardware costs
DL's reduction of the required feature/model engineering is a big deal for difficult problem domains where the cost of a mistake is low. That doesn't mean you don't still benefit from adding development resources, it's just that the development cost/performance tradeoff is still typically better than with a similar-performing explainable solution. I hope this will change in the future.
> ..., logistic regression with l1 penalization and only a little bit of feature engineering often does really well,...
While I agree that Lasso is far more explainable than DL, its explainability rapidly degrades as the useful feature dimension increases and it requires significant feature engineering for good performance on difficult problems.
Re: Why are we using black box models in AI when we don’t need to? (2019)
#130The main advantage of a blackbox ML solution is shorter development time to a useful performance level. Creating a transparent, explainable solution typically takes more time, more work, and a higher level of expertise to get to the same performance level. If the problem is complicated and the cost of a mistake is low, then your best approach today is likely to be blackbox. If the cost of a mistake is high, you shoul…
> The main advantage of a blackbox ML solution is shorter development time to a useful performance level. I think that's kinda true but kinda false. Its true that deep learning often makes feature engineering moot. however, a lot of deep learning projects takes machine learning engineers and applied scientists along with a host of other support engineers + hardware costs ( I seen people at work say: I only used a 16…
gradient boosted trees are black box