A Sober Look at Bayesian Neural Networks
jacobbuckman.com
A Sober Look at Bayesian Neural Networks
1–10 of 52 posts
Re: A Sober Look at Bayesian Neural Networks
#2The author starts with: P(A|B) = P(B|A)P(B)/P(A)
This is Bayes the wrong way around. The last part should be P(A)/P(B). I have no hand in the dispute (reading here for the first time about it), but not getting the basics right is not very convincing.
Re: A Sober Look at Bayesian Neural Networks
#3The author starts with: P(A|B) = P(B|A)P(B)/P(A) This is Bayes the wrong way around. The last part should be P(A)/P(B). I have no hand in the dispute (reading here for the first time about it), but not getting the basics right is not very convincing.
It's not "not getting the basics right", it's a simple typo. A thing like that shouldn't invalidate a whole article (unless you have skin in the game for the opponent argument).
Why does HN have a pattern of dismissing whole articles due to simple typos? It's as if we're so habituated to skim and do tldr-reading that our brain is working overdrive to find the slightest excuse not to have to do any type of reading beyond surface level.
Re: A Sober Look at Bayesian Neural Networks
#4The author starts with: P(A|B) = P(B|A)P(B)/P(A) This is Bayes the wrong way around. The last part should be P(A)/P(B). I have no hand in the dispute (reading here for the first time about it), but not getting the basics right is not very convincing.
It's not "not getting the basics right", it's a simple typo. A thing like that shouldn't invalidate a whole article (unless you have skin in the game for the opponent argument). Why does HN have a pattern of dismissing whole articles due to simple typos? It's as if we're so habituated to skim and do tldr-reading that our brain is working overdrive to find the slightest excuse not to have to do any type of reading bey…
> unless you have skin in the game for the opponent argument
I stated explicitly otherwise (I'm not even in research), something you certainly couldn't miss. I'm not quite sure, if your allegation backfires.
Especially in hot temper, people make errors. But just then they should avoid making an easy target. The authors site also has some problems with rendering math (some does, some shows still dollar signs) and this adds to a first impression of sloppyness. I t is his decision to go public - but then he has to take the consequences.
Re: A Sober Look at Bayesian Neural Networks
#5Earlier quoted context omitted.
It's not "not getting the basics right", it's a simple typo. A thing like that shouldn't invalidate a whole article (unless you have skin in the game for the opponent argument). Why does HN have a pattern of dismissing whole articles due to simple typos? It's as if we're so habituated to skim and do tldr-reading that our brain is working overdrive to find the slightest excuse not to have to do any type of reading bey…
> unless you have skin in the game for the opponent argument I stated explicitly otherwise (I'm not even in research), something you certainly couldn't miss. I'm not quite sure, if your allegation backfires. Especially in hot temper, people make errors. But just then they should avoid making an easy target. The authors site also has some problems with rendering math (some does, some shows still dollar signs) and this…
It's unfortunate the first equation was mistyped, one way to check is that the Bayes's rule can be derived from a rule in conditional probability:
P(A,B) = P(B,A)
P(A|B) * P(B) = P(B|A) * P(A)
The formulation in the blog post would say instead:
P(A|B) * P(A) = P(B|A) * P(B)
which does not really make sense.
Re: A Sober Look at Bayesian Neural Networks
#6The author starts with: P(A|B) = P(B|A)P(B)/P(A) This is Bayes the wrong way around. The last part should be P(A)/P(B). I have no hand in the dispute (reading here for the first time about it), but not getting the basics right is not very convincing.
It's not "not getting the basics right", it's a simple typo. A thing like that shouldn't invalidate a whole article (unless you have skin in the game for the opponent argument). Why does HN have a pattern of dismissing whole articles due to simple typos? It's as if we're so habituated to skim and do tldr-reading that our brain is working overdrive to find the slightest excuse not to have to do any type of reading bey…
I mean it’s kinda a big typo.
Re: A Sober Look at Bayesian Neural Networks
#7Earlier quoted context omitted.
It's not "not getting the basics right", it's a simple typo. A thing like that shouldn't invalidate a whole article (unless you have skin in the game for the opponent argument). Why does HN have a pattern of dismissing whole articles due to simple typos? It's as if we're so habituated to skim and do tldr-reading that our brain is working overdrive to find the slightest excuse not to have to do any type of reading bey…
I mean it’s kinda a big typo.
Not really, it's one of the most common mistakes I see in bayesian calculations. If the author was basing a lengthy series of calculations on that first step, it would be worse (but in this case the expression is quickly replaced by a corrected version for the classification discussion).
Re: A Sober Look at Bayesian Neural Networks
#8The author starts with: P(A|B) = P(B|A)P(B)/P(A) This is Bayes the wrong way around. The last part should be P(A)/P(B). I have no hand in the dispute (reading here for the first time about it), but not getting the basics right is not very convincing.
It's not "not getting the basics right", it's a simple typo. A thing like that shouldn't invalidate a whole article (unless you have skin in the game for the opponent argument). Why does HN have a pattern of dismissing whole articles due to simple typos? It's as if we're so habituated to skim and do tldr-reading that our brain is working overdrive to find the slightest excuse not to have to do any type of reading bey…
Critical and technical literature needs to be held to a standard.
When the rhetor introduces errors in the artifact, the rhetor's ethos with the audience is diminished.
The more fundamental the error (getting a basic equation wrong I guess?) the more trust you lose with a knowledgeable audience. If the author doesn't see that a fundamental issue was introduced, they may not have been expert enough to not introduce additional errors; the reader must spend more time double-checking the components of the argument rather than thinking about the argument itself.
If someone comes to this article as a novice in the topic and stores the error as a fact, they may end up at least confused when approaching it again in the future. HN tends to have an audience representing deep knowledge in many fields, who end up providing a thorough and varied set of quality filters. These quality filters are also really helpful to the novice who may otherwise miss the typo.
Re: A Sober Look at Bayesian Neural Networks
#9 The critique is about the importance of priors in BNN. In my humble understanding of Bayesian reasoning the argument to defend any prior is that with enought data the learning method converges to the real distribution, so if the result of any learning method depends heavily of any prior assumption then that assumption is crucial and in no way can it be taken randomly. On the other hand, it is well known that deep learning can learn any random model, so in the end I think all of this is about the bias-variance trade off. If your prior has an infinite number of adjustable parameters (zero bias) then the variance becomes infinite ( your result will depend and becomes equal to the training set).
So in practice one should choose the prior with the minimum number of parameters (bias) that shows a good learning performance on the available training set.
Anyway, trying to measure how a prior generalizes or not in BNN seems to be another way of thinking about bias-variance, if there is more than this, I would like to know.
Re: A Sober Look at Bayesian Neural Networks
#10Earlier quoted context omitted.
I mean it’s kinda a big typo.
Not really, it's one of the most common mistakes I see in bayesian calculations. If the author was basing a lengthy series of calculations on that first step, it would be worse (but in this case the expression is quickly replaced by a corrected version for the classification discussion).
Do you have data to support that it’s the most common mistake? It seems obvious to me from P(A|B) = P(A, B)/P(B) and P(A, B) = P(B|A) P(A)