A practical explanation of a Naive Bayes classifier
monkeylearn.com
A practical explanation of a Naive Bayes classifier
1–10 of 42 posts
Re: A practical explanation of a Naive Bayes classifier
#2Considering the relative ease of implementation, classification accuracy with smaller datasets, and computational efficiency of Naive Bayes classifiers, I am surprised that they are not mentioned as often as other machine learning competitors, such as random forest.
Are there major drawbacks to Naive Bayes classifiers? Is it just that they aren't as accurate on large datasets?
Re: A practical explanation of a Naive Bayes classifier
#3Considering the relative ease of implementation, classification accuracy with smaller datasets, and computational efficiency of Naive Bayes classifiers, I am surprised that they are not mentioned as often as other machine learning competitors, such as random forest. Are there major drawbacks to Naive Bayes classifiers? Is it just that they aren't as accurate on large datasets?
It doesn't have a good accuracy. I have yet to see a real-life dataset where it's better than to just call LogisticRegressionCV from scikit-learn. For bigger datasets, you may use vowpal wabbit or Fasttext. It may be a little bit slower for the training (but not so much), and as fast as LR for the training. What is the purpose of using an algorithm when another one is just better?
Re: A practical explanation of a Naive Bayes classifier
#4Considering the relative ease of implementation, classification accuracy with smaller datasets, and computational efficiency of Naive Bayes classifiers, I am surprised that they are not mentioned as often as other machine learning competitors, such as random forest. Are there major drawbacks to Naive Bayes classifiers? Is it just that they aren't as accurate on large datasets?
The (conditional) independence assumptions it makes are pretty strong and usually inaccurate. In the text example, consider "Steph Curry" and "Warriors." These are going to show up very frequently together, and both provide strong evidence that the topic is sports. However, they don't provide strongly independent evidence of sports. Supposing Curry announced he was making an investment in a local restaurant, the article would probably mention the Warriors just because it's Curry.
The problem is that seeing "Warriors" when you see "Steph Curry" shouldn't make you that much more confident that it's sports than just seeing "Steph Curry" alone. Just like seeing "University" with "Stanford" shouldn't change any inferences much from seeing "Stanford" alone. In naive bayes models, these are treated as independent pieces of evidence and that can lead to overconfidence and errors.
Naive bayes is a generative model, so in flipping bayes rule around to discriminate classes, you have to be sure your probability model is decent. In a discriminative model, you just go straight to learning the p(class | observations) and have no requirement for a decent model of p(observations). p(observations) is the kind of model that would have to know about "university" and "stanford" being likely to be observed together. Often, it's better to go straight to the discriminative model. In this case, logistic regression.
Re: A practical explanation of a Naive Bayes classifier
#5Considering the relative ease of implementation, classification accuracy with smaller datasets, and computational efficiency of Naive Bayes classifiers, I am surprised that they are not mentioned as often as other machine learning competitors, such as random forest. Are there major drawbacks to Naive Bayes classifiers? Is it just that they aren't as accurate on large datasets?
It doesn't have a good accuracy. I have yet to see a real-life dataset where it's better than to just call LogisticRegressionCV from scikit-learn. For bigger datasets, you may use vowpal wabbit or Fasttext. It may be a little bit slower for the training (but not so much), and as fast as LR for the training. What is the purpose of using an algorithm when another one is just better?
There's a reason for that. NB needs to fit more params than LR (and other linear discriminant learners), e.g. for binary classification, NB needs to fit 2*N params while LR just N.
Re: A practical explanation of a Naive Bayes classifier
#6The examples other people are using are fairly narrow. I would like to substantiate that text categorization via naive bayes classifier is surprisingly accurate and simple. This paper[1] uses ngrams and a simple out of place measure to compare articles against different verticals and often sees greater than 99% accuracy for relatively small blocks of text. The out of place measure also adds a penalty to features not found in the document, which helps establish the individuality for the category classification. Raw matching performance is also fairly impressive; A less naive implementation is also highly parallelizable.
Re: A practical explanation of a Naive Bayes classifier
#7Earlier quoted context omitted.
It doesn't have a good accuracy. I have yet to see a real-life dataset where it's better than to just call LogisticRegressionCV from scikit-learn. For bigger datasets, you may use vowpal wabbit or Fasttext. It may be a little bit slower for the training (but not so much), and as fast as LR for the training. What is the purpose of using an algorithm when another one is just better?
There's a reason for that. NB needs to fit more params than LR (and other linear discriminant learners), e.g. for binary classification, NB needs to fit 2*N params while LR just N.
Not true. Let's derive it. Here's NB for a binary classification:
p(c|w)*p(w) = p(w|c) * p(c)
p(c|w) = p(w|c) * p(c) / p(w)
p(c|w) = p(w|c) * p(c) / [sum_i p(w|c_i) * p(c_i)]
p(c_1|w) = p(w|c_1) * p(c_1) / [sum_i p(w|c_i) * p(c_i)]
p(c_1|w) = 1 / [sum_i p(w|c_i) * p(c_i) / p(w|c_1) / p(c_1]
p(c_1|w) = 1 / [1 + p(w|c_0) / p(w|c_1) * p(c_0) / p(c_1)]
p(w|c_0) / p(w|c_1) = prod_i p(w_i | c_0) / p(w_i | c_1)
This is a broader thing that a given graphical model with a given fixed parametrization can be generatively or discriminatively trained. They will end up learning different models, but that's because they make different assumptions, not because of different parametrizations.
[0] linear stuff = intercept + sum_i (coefficient of word_i)
Re: A practical explanation of a Naive Bayes classifier
#8Considering the relative ease of implementation, classification accuracy with smaller datasets, and computational efficiency of Naive Bayes classifiers, I am surprised that they are not mentioned as often as other machine learning competitors, such as random forest. Are there major drawbacks to Naive Bayes classifiers? Is it just that they aren't as accurate on large datasets?
Boosted decision trees account for >50% of all Kaggle winners. That is the real surprise, since people rarely talk about boosted decision trees.
Re: A practical explanation of a Naive Bayes classifier
#9A practical issue for Naive Bayes that also infects linear models is bias w.r.t. document length. Typically when you are detecting a rare, relatively compact class such as sports articles (or spam) you will tend to have a strongly negative prior, many positive features, and few negative ones. As a consequence, as the length of your text increases, not only does the variance of your prediction increase, but the mean tends to as well. This leads to all very long documents being classified as positive, regardless of their text. You can observe this by training your model and then classifying /usr/dict/words.
This is the most common mistake I've seen in production use of linear models on document text. Invariably, they'll misfire on any unusually long document.
Re: A practical explanation of a Naive Bayes classifier
#10https://github.com/bettinson/bayesian-tag-suggestion
I wrote one of these in Ruby to classify links into tags! Was fun. I think it got me a job.