A Sober Look at Bayesian Neural Networks • Hacker News Board

Re: A Sober Look at Bayesian Neural Networks

#11

Post by tw1010 » Sat, Jan 18, 2020, 2:13 PM UTC

toxik wrote: post #10

tw1010 wrote: post #7

Earlier quoted context omitted.

Not really, it's one of the most common mistakes I see in bayesian calculations. If the author was basing a lengthy series of calculations on that first step, it would be worse (but in this case the expression is quickly replaced by a corrected version for the classification discussion).

Do you have data to support that it’s the most common mistake? It seems obvious to me from P(A|B) = P(A, B)/P(B) and P(A, B) = P(B|A) P(A)

I didn't say it was one of the most common mistakes, I said it's one of the most common mistakes I have observed. Purely subjective.

Re: A Sober Look at Bayesian Neural Networks

#12

Post by nafizh » Sat, Jan 18, 2020, 2:19 PM UTC

Frankly, just solve an important problem with BNNs in a meaningful way with advantages that only BNNs bring (e.g uncertainty estimation) and you won’t have to write blogposts defending them.

Re: A Sober Look at Bayesian Neural Networks

#13

Post by cgel » Sat, Jan 18, 2020, 2:33 PM UTC

mbeex wrote: post #2

The author starts with: P(A|B) = P(B|A)P(B)/P(A) This is Bayes the wrong way around. The last part should be P(A)/P(B). I have no hand in the dispute (reading here for the first time about it), but not getting the basics right is not very convincing.

Author here. The blog post contains a clear explanation of Bayesian Inference and provides careful arguments about its potential benefits and limitations. Further, Bayes rule was typed correctly when it was being used to show that the posterior q(f^*|D) \approx q(f_\theta|D).

We understand Bayes rule... just had a typo and when proof reading the article we didn't check the first equation because: Who would mess up Bayes rule?

Re: A Sober Look at Bayesian Neural Networks

#14

Post by psv1 » Sat, Jan 18, 2020, 2:38 PM UTC

The paragraph beginning with "Let’s consider how we might apply the Bayesian framework..." where he introduces the notation is a great example of everything I hate about mathematical notation. We have big-F, small-f, f-of-x, f-sub-x, f-star, big-F-star... and then he decides to abbreviate what he just introduced. If I didn't know what's happening and I was trying to understand this for the first time, I would have no chance and would just give up right there.

Re: A Sober Look at Bayesian Neural Networks

#15

Post by macleginn » Sat, Jan 18, 2020, 2:42 PM UTC

tw1010 wrote: post #11

toxik wrote: post #10

Earlier quoted context omitted.

Do you have data to support that it’s the most common mistake? It seems obvious to me from P(A|B) = P(A, B)/P(B) and P(A, B) = P(B|A) P(A)

I didn't say it was one of the most common mistakes, I said it's one of the most common mistakes I have observed. Purely subjective.

I have a similar experience, but precisely because of this I am very careful to check the formula each time I have to type it. Mistakes in the formulas are not "just typos", they are a very annoying and potentially harmful kind of typos, and we must take great care in order to avoid them.

Re: A Sober Look at Bayesian Neural Networks

#16

Post by mlvljr » Sat, Jan 18, 2020, 2:53 PM UTC

psv1 wrote: post #14

The paragraph beginning with "Let’s consider how we might apply the Bayesian framework..." where he introduces the notation is a great example of everything I hate about mathematical notation. We have big-F, small-f, f-of-x, f-sub-x, f-star, big-F-star... and then he decides to abbreviate what he just introduced. If I didn't know what's happening and I was trying to understand this for the first time, I would have no…

Bif F stars all the way down ;)

Re: A Sober Look at Bayesian Neural Networks

#17

Post by tw1010 » Sat, Jan 18, 2020, 2:56 PM UTC

nafizh wrote: post #12

Frankly, just solve an important problem with BNNs in a meaningful way with advantages that only BNNs bring (e.g uncertainty estimation) and you won’t have to write blogposts defending them.

There's value in collaboratively discussing half-baked ideas in public before any real applications have been produced.

Imagine this article as just another in a stream of posts trying to "think aloud" about BNNs, without any immediate pressure of applications for it.

Re: A Sober Look at Bayesian Neural Networks

#18

Post by jph00 » Sat, Jan 18, 2020, 2:59 PM UTC

mattlutze wrote: post #8

tw1010 wrote: post #3

Earlier quoted context omitted.

It's not "not getting the basics right", it's a simple typo. A thing like that shouldn't invalidate a whole article (unless you have skin in the game for the opponent argument). Why does HN have a pattern of dismissing whole articles due to simple typos? It's as if we're so habituated to skim and do tldr-reading that our brain is working overdrive to find the slightest excuse not to have to do any type of reading bey…

Critical and technical literature needs to be held to a standard. When the rhetor introduces errors in the artifact, the rhetor's ethos with the audience is diminished. The more fundamental the error (getting a basic equation wrong I guess?) the more trust you lose with a knowledgeable audience. If the author doesn't see that a fundamental issue was introduced, they may not have been expert enough to not introduce ad…

> Critical and technical literature needs to be held to a standard.

That's a sloppy statement. You haven't defined what standard. Clearly, everything is held to "a standard"; making that an entirely empty claim.

> When the rhetor introduces errors in the artifact, the rhetor's ethos with the audience is diminished.

A "rhetor" is a teacher of rhetoric. This is not the correct word in this case. In this case, the correct word is the more general "author", since the post was not teaching rhetoric. Further, the entire point of "ethos" in rhetoric is that we shouldn't be so lazy as to allow minor issues cloud our judgement.

> The more fundamental the error (getting a basic equation wrong I guess?) the more trust you lose with a knowledgeable audience.

Quite the opposite. A knowledgeable audience can decide whether to trust something based on the actual content, rather than minor surface issues. Only a lazy or uninformed audience need get distracted by typos.

Re: A Sober Look at Bayesian Neural Networks

#19

Post by tw1010 » Sat, Jan 18, 2020, 2:59 PM UTC

macleginn wrote: post #15

tw1010 wrote: post #11

Earlier quoted context omitted.

I didn't say it was one of the most common mistakes, I said it's one of the most common mistakes I have observed. Purely subjective.

I have a similar experience, but precisely because of this I am very careful to check the formula each time I have to type it. Mistakes in the formulas are not "just typos", they are a very annoying and potentially harmful kind of typos, and we must take great care in order to avoid them.

Like I said above, "typos" in sequences of calculations are obviously problematic, and lead (almost always) to mistakes in the final result. In this case that's not applicable since there's no "second arithmetic manipulation" following the typo:ed one. (The author replaces the incorrect one with the correct bayesian equation in the next section.)

Re: A Sober Look at Bayesian Neural Networks

#20

Post by [deleted] » Sat, Jan 18, 2020, 3:02 PM UTC

psv1 wrote: post #14

The paragraph beginning with "Let’s consider how we might apply the Bayesian framework..." where he introduces the notation is a great example of everything I hate about mathematical notation. We have big-F, small-f, f-of-x, f-sub-x, f-star, big-F-star... and then he decides to abbreviate what he just introduced. If I didn't know what's happening and I was trying to understand this for the first time, I would have no…

[deleted]