Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Kagglers Are Using for Text Classification

mlwhiz.com

Post reply on HN
Print view View original

41–50 of 72 posts

Re: What Kagglers Are Using for Text Classification

#41

Post by briga »

elyase wrote: post #22

We have done extensive testing in the context of chatbot intent classification and in our particular problem nothing (including CNN, LSTM, fasttext plus LUIS, Watson and other proprietary classifiers) has been able to beat a simple linear model trained on char n-gram features.

I've seen the same things in the models I've built. For basic intent classification simpler models seem to be more accurate, not to mention they train faster and require less memory. There seems to be a lot of emphasis on shiny complex neural network architectures, even when simple models work just fine.

Re: What Kagglers Are Using for Text Classification

#42

Post by 995533 »

rundigen12 wrote: post #37
995533 wrote: post #23

Earlier quoted context omitted.

Nobody cares how long it takes to train a model. What matters is prediction speeds, which are comparable (and NLP less likely to require high frequency, where a few more milliseconds matters). Besides that, the accuracy gains are not marginal anymore (BoW can't compete like it used to, especially with pre-trained models).

> Nobody cares how long it takes to train a model. LOTS of people care how long it takes to train a model. A few minutes, vs. a day, vs. a week, vs. a month? Yea, that matters. Think about how long it takes to try out different hyperparameters or make other adjustments while conducting research... If you're Google maybe you don't care as much because you can fire off a hundred different jobs at once, but if you're a…

If you are building large-scale systems that take weeks or months to train, you are at a point where you shouldn't care about this. Throw more compute at the problem, it will pay for itself.

If we are talking days or hours: start parameter search on Friday and return best parameters on Monday.

Do research and iteration on heavily subsampled datasets.

If you are building models for yourself, or for Kaggle, you may care in as much as your laptop gets uncomfortably hot.

Re: What Kagglers Are Using for Text Classification

#43

Post by 995533 »

Cybiote wrote: post #34
995533 wrote: post #23

Earlier quoted context omitted.

Nobody cares how long it takes to train a model. What matters is prediction speeds, which are comparable (and NLP less likely to require high frequency, where a few more milliseconds matters). Besides that, the accuracy gains are not marginal anymore (BoW can't compete like it used to, especially with pre-trained models).

> Nobody cares how long it takes to train a model. This isn't true. It depends on your priorities and goals. Machine learning that spends most of its time unable to learn is not real AI. Some of us are interested in sample and energy efficient learning capable of on-line incremental updates immune to catastrophic forgetting. Not just because this is truer to actual learning but because it moves away from being depend…

Nobody in business cares if you are doing proper AI or dumb curve fitting. What matters is the complexity (engineering debt) and performance (accuracy, robustness).

Online learning, sample - and energy efficiency are unrelated to training times. Like said: nobody cares if you ran Vowpal Wabbit for 1 hour or 100 hours, as long as you are not constantly babysitting it and calling that paid work (or have the unusual requirement of daily retraining while using an online model).

> simple dot product is always going to be faster than many matrix multiplies

If you care about this (because it is profitable), you rewrite in lower-level language or predict with cloud GPU (which will be at least comparable to simple dot product, while adding performance)

Re: What Kagglers Are Using for Text Classification

#44

Post by thanatropism »

elyase wrote: post #22

We have done extensive testing in the context of chatbot intent classification and in our particular problem nothing (including CNN, LSTM, fasttext plus LUIS, Watson and other proprietary classifiers) has been able to beat a simple linear model trained on char n-gram features.

I've been curious about what manifold learning (sp. now that we have UMAP) would do to that kind of workflow's performance.

Re: What Kagglers Are Using for Text Classification

#45

Post by 995533 »

darkpuma wrote: post #29
995533 wrote: post #23

Earlier quoted context omitted.

Nobody cares how long it takes to train a model. What matters is prediction speeds, which are comparable (and NLP less likely to require high frequency, where a few more milliseconds matters). Besides that, the accuracy gains are not marginal anymore (BoW can't compete like it used to, especially with pre-trained models).

Time to train a model matters for applications where you want to have end users training models on their own computers without spending so much CPU/GPU time that they have to plan their day around it. Consider for instance an RSS reader that classifies articles to determine whether or not to interrupt the user with a notification. This should be fast to train and update the model on the fly every time the user enters…

I would not retrain such a model on all data, just do online updates. Also I still think for that use case training times and latency are negligible (nobody cares or nobody notices any difference between training a BoW and bi-LSTM.)

If you are deploying on resource-constrainted devices (IE: low-end PC's without GPU), it is not unusual to take a lot of time training a model on a very powerful computer (which nobody cares about), then distilling or transfering the result for test time.

Re: What Kagglers Are Using for Text Classification

#46

Post by Fomite »

995533 wrote: post #23
minimaxir wrote: post #14

It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.

Nobody cares how long it takes to train a model. What matters is prediction speeds, which are comparable (and NLP less likely to require high frequency, where a few more milliseconds matters). Besides that, the accuracy gains are not marginal anymore (BoW can't compete like it used to, especially with pre-trained models).

I recently had a very real world project be forced to abandon some promising methods because they were taking too long to train.

Re: What Kagglers Are Using for Text Classification

#48

Post by 995533 »

Fomite wrote: post #46
995533 wrote: post #23

Earlier quoted context omitted.

Nobody cares how long it takes to train a model. What matters is prediction speeds, which are comparable (and NLP less likely to require high frequency, where a few more milliseconds matters). Besides that, the accuracy gains are not marginal anymore (BoW can't compete like it used to, especially with pre-trained models).

I recently had a very real world project be forced to abandon some promising methods because they were taking too long to train.

It was not possible to increase speed by getting more powerful compute?

Re: What Kagglers Are Using for Text Classification

#49

Post by Fomite »

995533 wrote: post #48
Fomite wrote: post #46

Earlier quoted context omitted.

I recently had a very real world project be forced to abandon some promising methods because they were taking too long to train.

It was not possible to increase speed by getting more powerful compute?

No. Resources are not infinite, and we were already on the edge of what the resources at most sites where training would be done could be expected to have.

Re: What Kagglers Are Using for Text Classification

#50

Post by FridgeSeal »

briga wrote: post #41
elyase wrote: post #22

We have done extensive testing in the context of chatbot intent classification and in our particular problem nothing (including CNN, LSTM, fasttext plus LUIS, Watson and other proprietary classifiers) has been able to beat a simple linear model trained on char n-gram features.

I've seen the same things in the models I've built. For basic intent classification simpler models seem to be more accurate, not to mention they train faster and require less memory. There seems to be a lot of emphasis on shiny complex neural network architectures, even when simple models work just fine.

> There seems to be a lot of emphasis on shiny complex neural network architectures, even when simple models work just fine.

It's resume-driven-development for data scientists.

I've never seen an interviewer impressed with the fact that a job was performed using not-deep learning, but say that you used deep learning (despite how spurious it might be) and they light up like it's Christmas.

Post reply on HN