11–20 of 28 posts
Public accountability is designed into the algorithm at the outset.
Its funny, and characteristic of that part of the field, that mathematical provability in not even mentioned explicitly. A model (and a learning algorithm) should be interpretable if we can clearly state the assumptions, and prove that under these assumptions, we get what we state we get. We can predict the movement of the planets (short term). This is interpretable, because we only assume a model of 3d space, and Ne…
I don’t agree with this. For example, you can set up a collection of assumptions to underpin frequentist statistics, and then create a set of theorems about consistent or unbiased estimators, and develop a theory like that of p-values. But then in a practical setting, the model doesn’t correspond to something physical or to the inference goal of a practitioner. The p-value tells you something about the relative extre…
But if we do talk about that though. Yes, science proceeds by modifying or sometimes totally abandoning the assumptions. No silver bullet, and the point is to assume as little as possible.
In general, if things "just work", its not a reason to abandon attempts to understand them. Things "just work" until they don't, especially in finance.
Consider two guys. Guy A, he predicts that the sun rises every day and is there until dusk. It "works", and so, as the joke goes, he does not touch anything. And guy B, he knows physics and has telescopes. And he can predict the solar eclipses. He knows the failure modes of the model of guy A. Consider the difference between the two.
I think interpretability is much more correlated with model size than model type. Small neural net is much more interpretable than decision trees with thousands of nodes.
Earlier quoted context omitted.
I don’t agree with this. For example, you can set up a collection of assumptions to underpin frequentist statistics, and then create a set of theorems about consistent or unbiased estimators, and develop a theory like that of p-values. But then in a practical setting, the model doesn’t correspond to something physical or to the inference goal of a practitioner. The p-value tells you something about the relative extre…
I don't think there's a contradiction. My comment was about what it means to understand the model, or perhaps about what it means to do science, but not about what is necessary or sufficient for applied work. But if we do talk about that though. Yes, science proceeds by modifying or sometimes totally abandoning the assumptions. No silver bullet, and the point is to assume as little as possible. In general, if things…
If all that astronomy offered was a more verbose description of why the sun rises, yielding literally zero different predictions from Guy A’s super simple model, no one would care, and might call Guy B a witch!
I’d argue that what matters for science is pure predictive efficacy. That’s it. If you can explain something, it means you can accurately predict something about an unknown state of affairs that would falsify your model if your prediction is wrong. That’s it. Any other kind of explainability is just a matter of linguistic convenience.
Of course there is overfitting, etc., but that’s just part of refining the model to yield greater predictive accuracy on unseen data (less generalization error). It’s still all about putting your predictions where your mouth is.
If the differing theories of Guy A and Guy B cannot be separated by actually testing the predictions they make, then there is no “interpretability” — just linguistic hand waving.
Incidentally I think a lot of modern focus on interpretability is actually about how to be a political gatekeeper or taste-maker through linguistic hand waving, and is not about developing models who better survive the rigors of being required to make real predictions about states of affairs.
Its funny, and characteristic of that part of the field, that mathematical provability in not even mentioned explicitly. A model (and a learning algorithm) should be interpretable if we can clearly state the assumptions, and prove that under these assumptions, we get what we state we get. We can predict the movement of the planets (short term). This is interpretable, because we only assume a model of 3d space, and Ne…
I don’t agree with this. For example, you can set up a collection of assumptions to underpin frequentist statistics, and then create a set of theorems about consistent or unbiased estimators, and develop a theory like that of p-values. But then in a practical setting, the model doesn’t correspond to something physical or to the inference goal of a practitioner. The p-value tells you something about the relative extre…
How does this invalidate the notion of interpretability?
A model can be both wrong/misapplied and fully interpretable.
I think interpretability is much more correlated with model size than model type. Small neural net is much more interpretable than decision trees with thousands of nodes.
I challenge that last statement. Have there been any neural nets that actually have a solid interpretability? Usually those are more on the lines of effectiveness in training and validation data. With no real clue as to what the driving features were.
I work with random forests and build forest which have more than 80000 nodes per tree. Other than some basic computation of feature importance, it is a black box on the same scale as modern neural nets, maybe even worse.
Earlier quoted context omitted.
I don’t agree with this. For example, you can set up a collection of assumptions to underpin frequentist statistics, and then create a set of theorems about consistent or unbiased estimators, and develop a theory like that of p-values. But then in a practical setting, the model doesn’t correspond to something physical or to the inference goal of a practitioner. The p-value tells you something about the relative extre…
>But then in a practical setting, the model doesn’t correspond to something physical or to the inference goal of a practitioner. How does this invalidate the notion of interpretability? A model can be both wrong/misapplied and fully interpretable.
Who said that it did?
I’m saying the development of a model that yields scientific progress doesn’t have to have a connection to the parent comment’s proposed definition of interpretability, and may have competing interpretability concerns driven by specific inference goals that have nothing to do with proofs about the model under assumptions.
Earlier quoted context omitted.
>But then in a practical setting, the model doesn’t correspond to something physical or to the inference goal of a practitioner. How does this invalidate the notion of interpretability? A model can be both wrong/misapplied and fully interpretable.
> “How does this invalidate the notion of interpretability?” Who said that it did? I’m saying the development of a model that yields scientific progress doesn’t have to have a connection to the parent comment’s proposed definition of interpretability, and may have competing interpretability concerns driven by specific inference goals that have nothing to do with proofs about the model under assumptions.
Let's say someone used a neural network to predict planetary positions as a function of time, mass and so on. They made a network with several hundred parameters, trained until it stopped converging and published their final state vector.
Would that ever lead to gravitational theory?
Earlier quoted context omitted.
> “How does this invalidate the notion of interpretability?” Who said that it did? I’m saying the development of a model that yields scientific progress doesn’t have to have a connection to the parent comment’s proposed definition of interpretability, and may have competing interpretability concerns driven by specific inference goals that have nothing to do with proofs about the model under assumptions.
>I’m saying the development of a model that yields scientific progress doesn’t have to have a connection to the parent comment’s proposed definition of interpretability Let's say someone used a neural network to predict planetary positions as a function of time, mass and so on. They made a network with several hundred parameters, trained until it stopped converging and published their final state vector. Would that e…
Another point of view would be to say how does the Wiles proof of Fermat’s Last Theorem give us “understanding” (or substitute virtually any non-constructive existence proof, or proofs by contradiction, such as for the Halting Problem).
Compact math descriptions are good, don’t get me wrong. When the mechanism of some data generating process is actually oriented in such a way that we can get a concise math description then we absolutely should. But that is largely unrelated to advancing understanding or scientific progress, which can perfectly be some ugly enumerated garble of parameters, or some extremely piecewise solution space. Not every data generating process is required to admit a concise mathematical description.
Anyone have other good references for this topic?
https://www.youtube.com/watch?v=EZBUDG12Nr0&list=PLD63A284B7...