11–20 of 42 posts
https://monkeylearn.com/blog/how-to-create-text-classifiers-...
A practical issue for Naive Bayes that also infects linear models is bias w.r.t. document length. Typically when you are detecting a rare, relatively compact class such as sports articles (or spam) you will tend to have a strongly negative prior, many positive features, and few negative ones. As a consequence, as the length of your text increases, not only does the variance of your prediction increase, but the mean t…
Considering the relative ease of implementation, classification accuracy with smaller datasets, and computational efficiency of Naive Bayes classifiers, I am surprised that they are not mentioned as often as other machine learning competitors, such as random forest. Are there major drawbacks to Naive Bayes classifiers? Is it just that they aren't as accurate on large datasets?
Since naive bayes makes an independence assumption, and most competitions have features that are highly correlated, its probability / confidence of its prediction is probably a lot more useless than other classifiers.
besides that, nb doesn't do feature selection, regularization, and interactions like many tree based algorithms (such as XGBoost) do out of the box.
interactions is a big one for me. With NB, you have to add an interaction feature by multiplying two together, but in creating the feature you must steal some probability away from the other stuff, and finding a good interaction among many features is hard. The tree structure of random forests naturally captures that
edit: to your last point, nb does worse on larger data sets. when you dont have a lot of data, you have to make larger assumptions. NB makes a big independence assumption, thus it does well on very small data sets but falls short on large ones
A practical issue for Naive Bayes that also infects linear models is bias w.r.t. document length. Typically when you are detecting a rare, relatively compact class such as sports articles (or spam) you will tend to have a strongly negative prior, many positive features, and few negative ones. As a consequence, as the length of your text increases, not only does the variance of your prediction increase, but the mean t…
Is there some way to normalize the document length?
1. Use only the beginning of the document, as that's probably the most important part anyways, and it's fast.
2. Divide the sum of your feature scores by sqrt(n) to give it constant variance, and hopefully keep it comparable with your prior.
3. Split the doc into reasonably sized chunks, and average their scores rather than adding them.
The examples other people are using are fairly narrow. I would like to substantiate that text categorization via naive bayes classifier is surprisingly accurate and simple. This paper[1] uses ngrams and a simple out of place measure to compare articles against different verticals and often sees greater than 99% accuracy for relatively small blocks of text. The out of place measure also adds a penalty to features not…
(1) High dimensionality - this is also the reason why SVMs with linear kernels do so well on text (NBs find a linear boundary too - but a different linear boundary than linear-kernel SVMs)
(2) this reason applies to certain cases only: text where "token" based estimates are sufficiently discriminatory. For ex take a dataset that has two kinds of documents - one talking about product A and the other talking about product B. And you want to label the documents based on which product they're talking about. Here, just noticing if the term "A" or "B" appears in the document is good enough for classification. You don't have to have powerful models that infer "connotations" of words based on context. NB will do well here, esp if bundled with a feature selection technique that weeds out noisy features (like sthg based on Normalized Mutual Information)
I would add that another practical aspect about Naive Bayes classifiers is that you can make use of the conditional probabilities for each feature that contributes to the predictions. That gives you some introspection on how the model is working and it's useful when "debugging" classifiers by finding features that should/shouldn't be used. https://monkeylearn.com/blog/how-to-create-text-classifiers-...
I'm a total ML noob but it was a interesting project and the results were pretty accurate.
I basically used reddit's Bigquery data for the dataset (it's huge!). If you need a practical example of this algo, the algorithm and code is here[2].
[1] https://storage.googleapis.com/superasn/script.html
[2] https://www.reddit.com/r/learnmachinelearning/comments/6hqd6...
Considering the relative ease of implementation, classification accuracy with smaller datasets, and computational efficiency of Naive Bayes classifiers, I am surprised that they are not mentioned as often as other machine learning competitors, such as random forest. Are there major drawbacks to Naive Bayes classifiers? Is it just that they aren't as accurate on large datasets?
So, in cases:
(1) data cannot be separated linearly - NBs don't do well for the same reason as any other linear classifier wouldn't work well
(2) for linearly separable data - if the feature independence assumption is incompatible with the data, linear classifiers that don't make this assumption will beat it. For ex you can prove that logistic regression beats (or is at least as good as) Naive Bayes asymptotically. [1]
[1] See proposition 1 here: "On Discriminative vs Generative classifiers ... Andrew ng, Michael Jordan. https://ai.stanford.edu/~ang/papers/nips01-discriminativegen... [PDF]
A practical issue for Naive Bayes that also infects linear models is bias w.r.t. document length. Typically when you are detecting a rare, relatively compact class such as sports articles (or spam) you will tend to have a strongly negative prior, many positive features, and few negative ones. As a consequence, as the length of your text increases, not only does the variance of your prediction increase, but the mean t…
A practical issue for Naive Bayes that also infects linear models is bias w.r.t. document length. Typically when you are detecting a rare, relatively compact class such as sports articles (or spam) you will tend to have a strongly negative prior, many positive features, and few negative ones. As a consequence, as the length of your text increases, not only does the variance of your prediction increase, but the mean t…
Is there some way to normalize the document length?