Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Kagglers Are Using for Text Classification

mlwhiz.com

Post reply on HN
Print view View original

21–30 of 72 posts

Re: What Kagglers Are Using for Text Classification

#21

Post by soraki_soladead »

minimaxir wrote: post #14

It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.

For most businesses a GPU is not really expensive, particularly if the results are important to the business.

Chasing a single metric is certainly too narrow but getting the best results often does matter in a professional context too. While Kaggle can go overboard on massive ensembles using a state-of-the-art approach to the problem is often warranted outside of Kaggle.

Re: What Kagglers Are Using for Text Classification

#22

Post by elyase »

We have done extensive testing in the context of chatbot intent classification and in our particular problem nothing (including CNN, LSTM, fasttext plus LUIS, Watson and other proprietary classifiers) has been able to beat a simple linear model trained on char n-gram features.

Re: What Kagglers Are Using for Text Classification

#23

Post by 995533 »

minimaxir wrote: post #14

It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.

Nobody cares how long it takes to train a model. What matters is prediction speeds, which are comparable (and NLP less likely to require high frequency, where a few more milliseconds matters).

Besides that, the accuracy gains are not marginal anymore (BoW can't compete like it used to, especially with pre-trained models).

Re: What Kagglers Are Using for Text Classification

#24

Post by minimaxir »

elyase wrote: post #22

We have done extensive testing in the context of chatbot intent classification and in our particular problem nothing (including CNN, LSTM, fasttext plus LUIS, Watson and other proprietary classifiers) has been able to beat a simple linear model trained on char n-gram features.

Chatbot intent would be a good use case for a linear model, as a single word/ngram would have a high impact on the result (in contrast to advanced architectures which try to account for ambiguity/contradictions in documents)

Re: What Kagglers Are Using for Text Classification

#26

Post by eggie5 »

elyase wrote: post #22

We have done extensive testing in the context of chatbot intent classification and in our particular problem nothing (including CNN, LSTM, fasttext plus LUIS, Watson and other proprietary classifiers) has been able to beat a simple linear model trained on char n-gram features.

I've seen a rule of thumb that if the ratio of samples to words per sample is less than 1500 you prob don't have enough data for embeddings/cnn

Re: What Kagglers Are Using for Text Classification

#27

Post by alexcnwy »

minimaxir wrote: post #18
alexcnwy wrote: post #16

Earlier quoted context omitted.

CNNs are usually much faster than LSTMs.

Not related to the original post, but I recommend giving CuDNNLSTMs a try (if using Keras like the OP), the speed increase is absurd. (2-3x).

thanks for the tip!

Re: What Kagglers Are Using for Text Classification

#28

Post by laichzeit0 »

elyase wrote: post #22

We have done extensive testing in the context of chatbot intent classification and in our particular problem nothing (including CNN, LSTM, fasttext plus LUIS, Watson and other proprietary classifiers) has been able to beat a simple linear model trained on char n-gram features.

Anecdotally, I found the same thing. N-gram BOW is surprisingly difficult to beat.

Re: What Kagglers Are Using for Text Classification

#29

Post by darkpuma »

995533 wrote: post #23
minimaxir wrote: post #14

It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.

Nobody cares how long it takes to train a model. What matters is prediction speeds, which are comparable (and NLP less likely to require high frequency, where a few more milliseconds matters). Besides that, the accuracy gains are not marginal anymore (BoW can't compete like it used to, especially with pre-trained models).

Time to train a model matters for applications where you want to have end users training models on their own computers without spending so much CPU/GPU time that they have to plan their day around it.

Consider for instance an RSS reader that classifies articles to determine whether or not to interrupt the user with a notification. This should be fast to train and update the model on the fly every time the user enters a correction (e.g. 'this article actually isn't interesting', or 'interrupt me with articles like this in the future'.)

Re: What Kagglers Are Using for Text Classification

#30

Post by soraki_soladead »

biomodel wrote: post #11

Not sure why anyone would use 2D CNNs for processing text when there is no spatial correlation in the embedding features. Recent work such as https://arxiv.org/abs/1803.01271 show that for most tasks, 1D CNNs outperform recurrent architectures while being faster to train

This is just a bug in their code. The paper they cite uses 1D convolutions. Though, I suppose having an unused dimension only really hurts efficiency.
Post reply on HN