Not sure why anyone would use 2D CNNs for processing text when there is no spatial correlation in the embedding features. Recent work such as https://arxiv.org/abs/1803.01271 show that for most tasks, 1D CNNs outperform recurrent architectures while being faster to train
What Kagglers Are Using for Text Classification
11–20 of 72 posts
Re: What Kagglers Are Using for Text Classification
#12Three methods and no idea how I would choose between the three of them aside randomly trying each one and measuring performance. (Not for winning kaggle but for an actual problem)
[deleted]
Re: What Kagglers Are Using for Text Classification
#13It would be nice to see how these methods compare to the classical methods based on word occurrences.
Kaggle is a pretty serious natural-selection environment for machine learning algorithms. Basically, if bag-of-words worked better, the contest winners would still use it.
You're assuming that they aren't trying: it's not too hard to try out bag-of-words and see what happens. But things like Attention and LSTMs are pretty good, but not without their costs.
Re: What Kagglers Are Using for Text Classification
#14It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice.
Kaggle prioritizes chasing a metric, but real-world data science has more considerations.
Re: What Kagglers Are Using for Text Classification
#15Three methods and no idea how I would choose between the three of them aside randomly trying each one and measuring performance. (Not for winning kaggle but for an actual problem)
this is unfortunately almost always true, you just have to try every possible combination of everything with lots of hyperparameters. nothing makes any sense, it's total chaos, and we are wandering blind in the wastelands.
Re: What Kagglers Are Using for Text Classification
#16It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.
CNNs are usually much faster than LSTMs.
Re: What Kagglers Are Using for Text Classification
#17Three methods and no idea how I would choose between the three of them aside randomly trying each one and measuring performance. (Not for winning kaggle but for an actual problem)
this is unfortunately almost always true, you just have to try every possible combination of everything with lots of hyperparameters. nothing makes any sense, it's total chaos, and we are wandering blind in the wastelands.
minimaxir's other comment was helpful by not trying to guess an outcome but explaining things we do know about, like training/estimation cost and order of complexity, which were topics absent from the article
Re: What Kagglers Are Using for Text Classification
#18It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.
CNNs are usually much faster than LSTMs.
Not related to the original post, but I recommend giving CuDNNLSTMs a try (if using Keras like the OP), the speed increase is absurd. (2-3x).
Re: What Kagglers Are Using for Text Classification
#19Three methods and no idea how I would choose between the three of them aside randomly trying each one and measuring performance. (Not for winning kaggle but for an actual problem)
"The information you'd need to choose is included in there. If you're doing this professionally, you should strive to have enough of a high-level understanding of NLP to be able to make these decisions without having a rubric handed to you on a silver platter.
In a nutshell, though: Strive to use the simplest model that will get the job done. Less elaborate models are easier to understand and (usually) less prone to things like overfitting, so they'll be more tractable to work with in a business context. To that end: Use a convolutional net when you can get away with a small, fixed-size context window. Use an LSTM when you need long-term memory. Attention can be expensive, so you use it when you have cause to believe you can gain a lot by giving selective attention to features, and have both a lot of training data and a lot of computing resources.
It's also worth considering that you might be best off going with none of these options. Cool as deep learning is, I've personally never actually been able to justify using it in a professional setting. Simpler models such as logistic regression and decision trees have characteristics that are near-useless for getting you to the top of a Kaggle leaderboard, but can be indispensable when working on many real-world business problems"
- anonymous comment reply
Re: What Kagglers Are Using for Text Classification
#20Three methods and no idea how I would choose between the three of them aside randomly trying each one and measuring performance. (Not for winning kaggle but for an actual problem)
"The information you'd need to choose is included in there. If you're doing this professionally, you should strive to have enough of a high-level understanding of NLP to be able to make these decisions without having a rubric handed to you on a silver platter. In a nutshell, though: Strive to use the simplest model that will get the job done. Less elaborate models are easier to understand and (usually) less prone to…
This is the kind of context that is very helpful.