Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A practical explanation of a Naive Bayes classifier

monkeylearn.com

Post reply on HN
Print view View original

21–30 of 42 posts

Re: A practical explanation of a Naive Bayes classifier

#21

Post by moultano »

Houshalter wrote: post #20
intune wrote: post #12

Earlier quoted context omitted.

Is there some way to normalize the document length?

You can use term frequency instead of binary features. This is invariant to the size of the document. This is called multinomial naive Bayes: https://en.m.wikipedia.org/wiki/Naive_Bayes_classifier#Multi...

This is not invariant to the size of the document (though agreed, generally better). It doesn't solve the problem of having mostly positive features and a negative prior.

Stated more formally, your model is b + wᵀx. Generally, b is 0. As the document grows, wᵀx tends to dominate b. You'll have bias with length as long as E[wᵀx]≠0 and there aren't any constraints on w that would force this.

Re: A practical explanation of a Naive Bayes classifier

#23

Post by abhgh »

imh wrote: post #7
boto3 wrote: post #5

Earlier quoted context omitted.

There's a reason for that. NB needs to fit more params than LR (and other linear discriminant learners), e.g. for binary classification, NB needs to fit 2*N params while LR just N.

Not true. Let's derive it. Here's NB for a binary classification: p(c|w)*p(w) = p(w|c) * p(c) p(c|w) = p(w|c) * p(c) / p(w) p(c|w) = p(w|c) * p(c) / [sum_i p(w|c_i) * p(c_i)] Let's look at the probability of class 1. p(c_1|w) = p(w|c_1) * p(c_1) / [sum_i p(w|c_i) * p(c_i)] Notice how the numerator is going to show up in the denominator. We can simplify that by bringing it into the denominator: p(c_1|w) = 1 / [sum_i p…

I may be missing something, but how do you take a log on the RHS of p(c_1|w)? The denominator has a sum i.e. "1+ something", which the log doesn't distribute over.

Re: A practical explanation of a Naive Bayes classifier

#24

Post by abhgh »

j2kun wrote: post #8
schuetze wrote: post #2

Considering the relative ease of implementation, classification accuracy with smaller datasets, and computational efficiency of Naive Bayes classifiers, I am surprised that they are not mentioned as often as other machine learning competitors, such as random forest. Are there major drawbacks to Naive Bayes classifiers? Is it just that they aren't as accurate on large datasets?

Boosted decision trees account for >50% of all Kaggle winners. That is the real surprise, since people rarely talk about boosted decision trees.

Deep Learning is in vogue so I think we see a lot around it written about in blogs, media etc. GBDTs are used in the industry though.

Re: A practical explanation of a Naive Bayes classifier

#25

Post by b_ttercup »

Is Naive Bayes really ever the most practical choice? Yes it is a simple, fast algorithm, but it's usually a non trivial step below other simple models in my experience and doesn't seem to show any major advantages. The results shown here seem good but bag of words models usually do better than you might think on supervised NLP. So what's the motivation?

Re: A practical explanation of a Naive Bayes classifier

#26

Post by tyingq »

b_ttercup wrote: post #25

Is Naive Bayes really ever the most practical choice? Yes it is a simple, fast algorithm, but it's usually a non trivial step below other simple models in my experience and doesn't seem to show any major advantages. The results shown here seem good but bag of words models usually do better than you might think on supervised NLP. So what's the motivation?

I thought it was the typical approach for identifying email spam. Has that changed?

Re: A practical explanation of a Naive Bayes classifier

#27

Post by imh »

abhgh wrote: post #23
imh wrote: post #7

Earlier quoted context omitted.

Not true. Let's derive it. Here's NB for a binary classification: p(c|w)*p(w) = p(w|c) * p(c) p(c|w) = p(w|c) * p(c) / p(w) p(c|w) = p(w|c) * p(c) / [sum_i p(w|c_i) * p(c_i)] Let's look at the probability of class 1. p(c_1|w) = p(w|c_1) * p(c_1) / [sum_i p(w|c_i) * p(c_i)] Notice how the numerator is going to show up in the denominator. We can simplify that by bringing it into the denominator: p(c_1|w) = 1 / [sum_i p…

I may be missing something, but how do you take a log on the RHS of p(c_1|w)? The denominator has a sum i.e. "1+ something", which the log doesn't distribute over.

Heh, nope you're right I was writing to quickly. I meant to say to take a log of of the probability term in p(c_1|w). So you take p(c_1|w) = 1/(1+stuff) and turn it into p(c_1|w) = 1/(1+exp(log(stuff))). stuff is a product, so log(stuff) is a sum. Good catch.

Re: A practical explanation of a Naive Bayes classifier

#28

Post by _dps »

moultano wrote: post #14
intune wrote: post #12

Earlier quoted context omitted.

Is there some way to normalize the document length?

Lots of reasonable hacks. 1. Use only the beginning of the document, as that's probably the most important part anyways, and it's fast. 2. Divide the sum of your feature scores by sqrt(n) to give it constant variance, and hopefully keep it comparable with your prior. 3. Split the doc into reasonably sized chunks, and average their scores rather than adding them.

I'll add to this that you can add a very crude (separate) model for the document length and number of distinct words, and use that to flag outlier documents that might bump into the known weaknesses with respect to document length.

Re: A practical explanation of a Naive Bayes classifier

#29

Post by Houshalter »

moultano wrote: post #21
Houshalter wrote: post #20

Earlier quoted context omitted.

You can use term frequency instead of binary features. This is invariant to the size of the document. This is called multinomial naive Bayes: https://en.m.wikipedia.org/wiki/Naive_Bayes_classifier#Multi...

This is not invariant to the size of the document (though agreed, generally better). It doesn't solve the problem of having mostly positive features and a negative prior. Stated more formally, your model is b + wᵀx. Generally, b is 0. As the document grows, wᵀx tends to dominate b. You'll have bias with length as long as E[wᵀx]≠0 and there aren't any constraints on w that would force this.

If your data obeys the naive Bayes assumptions then this model is mathematically optimal. That each word is independently drawn from some distribution conditional on it's class. E.g. if there was an exactly 1% chance any given word in a spam email would be "viagra".

Now obviously real world data doesn't obey these assumptions perfectly. But I don't see how violating the independent features assumption would cause the problem you mention. A longer email does mean the word "viagra" is more likely to occur in a normal email just by random chance. But the model takes that into account by recording the frequency of "viagra" in normal emails and seeing if it's consistent with that.

Post reply on HN