Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Kagglers Are Using for Text Classification

mlwhiz.com

Post reply on HN
Print view View original

31–40 of 72 posts

Re: What Kagglers Are Using for Text Classification

#31

Post by linalgmixer »

wenc wrote: post #7

I wonder how FastText (essentially word2vec + word & char n-grams + other stuff) stacks up against these algorithms. In my own tests on my own corpuses, CPU-based FastText is faster to train and produces significantly better results (precision/recall) than the GPU-bound CNN algorithms that I've tried, but have not compared it against RNN techniques.

FastText is essentially a linear classifier and it's not surprising that it would train quickly. As for the prediction metrics, I imagine that will depend on the type of data and problem you're working with. Linear models perform very well on certain types of problems (I've had great success with SVMs on text classification problems personally) but for more complicated tasks, I imagine that the deep learning models would perform very, very well (relatively).

Re: What Kagglers Are Using for Text Classification

#32

Post by rundigen12 »

I was really hoping to see a summary comparison of the performance(s) of the different models at the end, e.g. accuracy vs. complexity vs. execution time, etc.

Here's a summary from the end of each section...

1. TextCNN: "This kernel scored around 0.661 on the public leaderboard."

2. BiDirectional RNN: 0.671

3. Attention Models: 0.682

Re: What Kagglers Are Using for Text Classification

#33

Post by physicsyogi »

wenc wrote: post #7

I wonder how FastText (essentially word2vec + word & char n-grams + other stuff) stacks up against these algorithms. In my own tests on my own corpuses, CPU-based FastText is faster to train and produces significantly better results (precision/recall) than the GPU-bound CNN algorithms that I've tried, but have not compared it against RNN techniques.

I’ve found that some CNNs consistently beat fasttext in terms of model quality. But I’ve beaten those CNNs and fasttext by doing transfer learning with ULMFit and fasts I. But if we’re talking training speed, fasttext is indeed aptly named.

Re: What Kagglers Are Using for Text Classification

#34

Post by Cybiote »

995533 wrote: post #23
minimaxir wrote: post #14

It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.

Nobody cares how long it takes to train a model. What matters is prediction speeds, which are comparable (and NLP less likely to require high frequency, where a few more milliseconds matters). Besides that, the accuracy gains are not marginal anymore (BoW can't compete like it used to, especially with pre-trained models).

> Nobody cares how long it takes to train a model.

This isn't true. It depends on your priorities and goals. Machine learning that spends most of its time unable to learn is not real AI. Some of us are interested in sample and energy efficient learning capable of on-line incremental updates immune to catastrophic forgetting. Not just because this is truer to actual learning but because it moves away from being dependent on a handful of companies to do the actual training.

Anticipating some replies: no, transfer learning or meta-learning methods don't really avoid this. In the case of transfer learning, you still have that high coupling between a handful of sources. The down-sides of this is its own discussion. In addition, there are times where the ability to extract local relations can be dulled by the dominant wikipedia and common-crawl representations. Meta-learning gets you fast updates but you still cannot stray too far away from the domains that were met at training time.

> What matters is prediction speeds

I'm not a fan of bag of words models either but a simple dot product is always going to be faster than many matrix multiplies and or convolutions. The implementor should always try these as a base-line and decide if the performance accuracy trade-off is worth it for them.

Re: What Kagglers Are Using for Text Classification

#35

Post by nerdponx »

minimaxir wrote: post #13
nerdponx wrote: post #5

Earlier quoted context omitted.

Kaggle is a pretty serious natural-selection environment for machine learning algorithms. Basically, if bag-of-words worked better, the contest winners would still use it.

You're assuming that they aren't trying: it's not too hard to try out bag-of-words and see what happens. But things like Attention and LSTMs are pretty good, but not without their costs.

I'm assuming the opposite: bag-of-words is the go-to baseline, but as another commenter pointed out, just separating instances based on vocabulary is not sufficient in today's sophisticated text classification problems.

Re: What Kagglers Are Using for Text Classification

#36

Post by nerdponx »

PaulHoule wrote: post #9
nerdponx wrote: post #5

Earlier quoted context omitted.

Kaggle is a pretty serious natural-selection environment for machine learning algorithms. Basically, if bag-of-words worked better, the contest winners would still use it.

One issue is the kind of problem. I remember getting 95%-ish accuracy with BoW and the SVM circa 2004 when it came to questions like "is this paper about astrophysics or organic chemistry?" In that case you have a distinct vocabulary for different topics and it is hard to beat BoW. Sentiment analysis, on the other hand, is where BoW goes to die since now "not good" means something very different than "good", and even…

Good point, I should have written "for machine learning algorithms on problems that the industry is currently interested in".

Re: What Kagglers Are Using for Text Classification

#37

Post by rundigen12 »

995533 wrote: post #23
minimaxir wrote: post #14

It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.

Nobody cares how long it takes to train a model. What matters is prediction speeds, which are comparable (and NLP less likely to require high frequency, where a few more milliseconds matters). Besides that, the accuracy gains are not marginal anymore (BoW can't compete like it used to, especially with pre-trained models).

> Nobody cares how long it takes to train a model.

LOTS of people care how long it takes to train a model. A few minutes, vs. a day, vs. a week, vs. a month? Yea, that matters.

Think about how long it takes to try out different hyperparameters or make other adjustments while conducting research...

If you're Google maybe you don't care as much because you can fire off a hundred different jobs at once, but if you're a resource-limited mere mortal, yea, that wait time adds up.

Re: What Kagglers Are Using for Text Classification

#38

Post by [deleted] »

Cybiote wrote: post #34
995533 wrote: post #23

Earlier quoted context omitted.

Nobody cares how long it takes to train a model. What matters is prediction speeds, which are comparable (and NLP less likely to require high frequency, where a few more milliseconds matters). Besides that, the accuracy gains are not marginal anymore (BoW can't compete like it used to, especially with pre-trained models).

> Nobody cares how long it takes to train a model. This isn't true. It depends on your priorities and goals. Machine learning that spends most of its time unable to learn is not real AI. Some of us are interested in sample and energy efficient learning capable of on-line incremental updates immune to catastrophic forgetting. Not just because this is truer to actual learning but because it moves away from being depend…

[deleted]

Re: What Kagglers Are Using for Text Classification

#39

Post by [deleted] »

minimaxir wrote: post #14

It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.

[deleted]

Re: What Kagglers Are Using for Text Classification

#40

Post by computerex »

minimaxir wrote: post #14

It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.

Agreed. For many many applications, bag of words/tf-idf/support vector machines are enough but with much better training time.
Post reply on HN