I thought this was a good post!
In fact, I had just read a CVPR '18 paper that did the kind of thing he mentioned--presented an 'interpretable neural network' that assumed what interpretability meant...
The Mythos of Model Interpretability in Machine Learning
21–28 of 28 posts
Re: The Mythos of Model Interpretability in Machine Learning
#22Earlier quoted context omitted.
I challenge that last statement. Have there been any neural nets that actually have a solid interpretability? Usually those are more on the lines of effectiveness in training and validation data. With no real clue as to what the driving features were.
https://distill.pub/2018/building-blocks/ This seems pretty good to me. And the nets are not exactly as small as I meant. I work with random forests and build forest which have more than 80000 nodes per tree. Other than some basic computation of feature importance, it is a black box on the same scale as modern neural nets, maybe even worse.
Thanks for the link. Will take me some time to digest it. Last time I looked at one of these, it was less informative than I'd care to admit. I want to know why a classifier found a vase. The answer is typically some form of "because it was able to see the vase."
Would be more interesting to see a classifier that predicts something is going to happen. A predictor that can predict a person is about to step off of a curb, for example. Is it pose of the person? Did it require seeing multiple frames of the person, such that it was an inertia preditor?
So, yes, comparing things to gigantic trees can make things tough. But I thought that was the beauty of boosting and building up smaller trees. Most of them are usually more interpretable than you might expect.
Re: The Mythos of Model Interpretability in Machine Learning
#23Earlier quoted context omitted.
>I’m saying the development of a model that yields scientific progress doesn’t have to have a connection to the parent comment’s proposed definition of interpretability Let's say someone used a neural network to predict planetary positions as a function of time, mass and so on. They made a network with several hundred parameters, trained until it stopped converging and published their final state vector. Would that e…
What do you mean by “gravitational theory” apart from “correctly accounts for predicting the gravitational effects that we observe”? If a neural net can do that, then yes, a perfectly “elegant” theory could be the giant enumeration of a big ugly bunch of parameters. Of course we might suspect that’s overfitting and whittle down to a compact description, and that’s great, but doesn’t make the ugly enumeration of a big…
>What do you mean by “gravitational theory” apart from “correctly accounts for predicting the gravitational effects that we observe”?
F = G (m1 * m2) / r^2
Newton's theory of gravitation identified the relevant variables and their relationships. It had no junk variables or parameters. You can do algebraic transformations on it and use the formula outside of the original context. It was not fully correct, but it made exact predictions. People were able to notice the discrepancies it had with orbits of some celestial bodies, which helped in identifying general theory of relativity.
All of this would have been lost of the formula was a state vector of a large neural network.
Re: The Mythos of Model Interpretability in Machine Learning
#24Earlier quoted context omitted.
What do you mean by “gravitational theory” apart from “correctly accounts for predicting the gravitational effects that we observe”? If a neural net can do that, then yes, a perfectly “elegant” theory could be the giant enumeration of a big ugly bunch of parameters. Of course we might suspect that’s overfitting and whittle down to a compact description, and that’s great, but doesn’t make the ugly enumeration of a big…
>What do you mean by “gravitational theory” apart from “correctly accounts for predicting the gravitational effects that we observe”? F = G (m1 * m2) / r^2 Newton's theory of gravitation identified the relevant variables and their relationships. It had no junk variables or parameters. You can do algebraic transformations on it and use the formula outside of the original context. It was not fully correct, but it made…
> “Newton's theory of gravitation identified the relevant variables and their relationships.”
No, it introduced a useful low-dimensional fiction that approximately works, with high accuracy at coarse scales, and doesn’t account for relativistic or quantum effects (and so, in an absolute sense, is only an approximation, not an interpretation of what is real).
Besides, what you glibly type out as a few characters of a formula unpackages into a deeply hard-to-explain theory in natural language. It’s vastly less interpretable than simpler (and less correct) models because it introduces much more complexity of the meaning and physical concepts behind the formula terms. Just as quantum or relativistic extensions add even more complexity and counter-intuitive concepts and become even harder to interpret.
> “All of this would have been lost of the formula was a state vector of a large neural network.”
Why? I see no reason why that would be true. If there is a very low-dimensional manifold in parameter space that accounts for all the predictive power, that would be a very easy thing to diagnose from a neural network model. Lots of things have been discovered that way, like graph centrality metrics coming from eigen decompositions, style transfer, glove vectors, eigenfaces.
Getting the high-dimensional representation that yields accurate predictions is often step 1, great scientific progress and often a huge leap forward. Then step 2 is often asking if there is some natural low-dimensional compression of the parameters that is approximately as good, and if so, does it correspond to any known quantities.
Though sometimes this isn’t possible or even desirable, such as with non-parametric models, and yet such models can still represent “interpretable” scientific progress in the sense of allowing us to predict how states of affairs will develop when we previously couldn’t and solving inference goals.
Re: The Mythos of Model Interpretability in Machine Learning
#25Earlier quoted context omitted.
https://distill.pub/2018/building-blocks/ This seems pretty good to me. And the nets are not exactly as small as I meant. I work with random forests and build forest which have more than 80000 nodes per tree. Other than some basic computation of feature importance, it is a black box on the same scale as modern neural nets, maybe even worse.
Thanks for the link. Will take me some time to digest it. Last time I looked at one of these, it was less informative than I'd care to admit. I want to know why a classifier found a vase. The answer is typically some form of "because it was able to see the vase." Would be more interesting to see a classifier that predicts something is going to happen. A predictor that can predict a person is about to step off of a cu…
> So, yes, comparing things to gigantic trees can make things tough. But I thought that was the beauty of boosting and building up smaller trees. Most of them are usually more interpretable than you might expect.
Regarding your last paragraph, I found this paper https://arxiv.org/pdf/1504.07676.pdf worth reading. Excerpt from the abstract: "We conclude that boosting should be used like random forests: with large decision trees and without direct regularization or early stopping."
Re: The Mythos of Model Interpretability in Machine Learning
#26Earlier quoted context omitted.
>What do you mean by “gravitational theory” apart from “correctly accounts for predicting the gravitational effects that we observe”? F = G (m1 * m2) / r^2 Newton's theory of gravitation identified the relevant variables and their relationships. It had no junk variables or parameters. You can do algebraic transformations on it and use the formula outside of the original context. It was not fully correct, but it made…
> “Newton's theory of gravitation identified the relevant variables and their relationships.” No, it introduced a useful low-dimensional fiction that approximately works, with high accuracy at coarse scales, and doesn’t account for relativistic or quantum effects (and so, in an absolute sense, is only an approximation, not an interpretation of what is real ). Besides, what you glibly type out as a few characters of a…
Science (and especially mathematics) is all about increasing human understanding. A neural network that makes correct predictions would be very interesting to both engineers and scientists.
From a scientific perspective, the fact something can be modeled well using a neural network would provide interesting constraints on the system, which may eventually yield breakthroughs in understanding the original system. However, the neural network's parameterized equation by itself would never be accepted as good science, since its complexity would hardly contribute anything to our understanding.
Re: The Mythos of Model Interpretability in Machine Learning
#27Earlier quoted context omitted.
>What do you mean by “gravitational theory” apart from “correctly accounts for predicting the gravitational effects that we observe”? F = G (m1 * m2) / r^2 Newton's theory of gravitation identified the relevant variables and their relationships. It had no junk variables or parameters. You can do algebraic transformations on it and use the formula outside of the original context. It was not fully correct, but it made…
> “Newton's theory of gravitation identified the relevant variables and their relationships.” No, it introduced a useful low-dimensional fiction that approximately works, with high accuracy at coarse scales, and doesn’t account for relativistic or quantum effects (and so, in an absolute sense, is only an approximation, not an interpretation of what is real ). Besides, what you glibly type out as a few characters of a…
> If there is a very low-dimensional manifold in parameter space that accounts for all the predictive power, that would be a very easy thing to diagnose from a neural network model.
People wouldn't waste time researching neural network visualization tools, coming up with adversarial examples and developing alternative (more explainable) machine learning models if that was the case.
Re: The Mythos of Model Interpretability in Machine Learning
#28I think interpretability is much more correlated with model size than model type. Small neural net is much more interpretable than decision trees with thousands of nodes.
I challenge that last statement. Have there been any neural nets that actually have a solid interpretability? Usually those are more on the lines of effectiveness in training and validation data. With no real clue as to what the driving features were.
> Small neural net is much more interpretable than decision trees with thousands of nodes.
Yes. Example a simple logistic regression model