Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Kagglers Are Using for Text Classification

mlwhiz.com

Post reply on HN
Print view View original

51–60 of 72 posts

Re: What Kagglers Are Using for Text Classification

#51

Post by 995533 »

Fomite wrote: post #49
995533 wrote: post #48

Earlier quoted context omitted.

It was not possible to increase speed by getting more powerful compute?

No. Resources are not infinite, and we were already on the edge of what the resources at most sites where training would be done could be expected to have.

Thanks. I think you are a correct exception to what I said. I should have known that using words like "nobody" would not go over well on HN (but tedious to type "a very large percentage"), despite that statement being a verbatim quote from one of the world's leading ML engineers and, to me, not controversial.

I do consider the cloud both widely available and near infinite in resource adding capability.

If it is really not economically feasible to add resources, then the performance gains were not as promising as thought (whether cloud or on-site).

Re: What Kagglers Are Using for Text Classification

#52

Post by Cybiote »

995533 wrote: post #43
Cybiote wrote: post #34

Earlier quoted context omitted.

> Nobody cares how long it takes to train a model. This isn't true. It depends on your priorities and goals. Machine learning that spends most of its time unable to learn is not real AI. Some of us are interested in sample and energy efficient learning capable of on-line incremental updates immune to catastrophic forgetting. Not just because this is truer to actual learning but because it moves away from being depend…

Nobody in business cares if you are doing proper AI or dumb curve fitting. What matters is the complexity (engineering debt) and performance (accuracy, robustness). Online learning, sample - and energy efficiency are unrelated to training times. Like said: nobody cares if you ran Vowpal Wabbit for 1 hour or 100 hours, as long as you are not constantly babysitting it and calling that paid work (or have the unusual req…

You've clarified your stance from nobody to nobody in business. That's good, although, I think that is opinion based on your experiences. I suspect that business will care if researchers can make it easy to learn on premise on their small datasets while maintaining high accuracy. The ability to easily update and adapt under non-stationarity without having to retrain from scratch benefits all. The same is true of models that maintain uncertainty or that can explain decision outputs. Tracking uncertainty, robustness to changes, on-line updatability and explainability are all related in that they are examples of things that become easier under causal modeling.

A parallel discussion we are having is whether the gain in accuracy is always worth the gain in complexity and loss in speed. It's something to decide on a case by case basis. It's basic hygiene to reach for the simplest model first.

Re: What Kagglers Are Using for Text Classification

#54

Post by madavidj »

biomodel wrote: post #11

Not sure why anyone would use 2D CNNs for processing text when there is no spatial correlation in the embedding features. Recent work such as https://arxiv.org/abs/1803.01271 show that for most tasks, 1D CNNs outperform recurrent architectures while being faster to train

Probably because the author followed this blog: http://www.wildml.com/2015/12/implementing-a-cnn-for-text-cl...

That blog used a 2d cnn because tensorflow didn't have a 1d version at the time of writing, so he just created a dummy 2nd dimension of length 1 and called it a day.

Re: What Kagglers Are Using for Text Classification

#55

Post by gnulinux »

soraki_soladead wrote: post #30
biomodel wrote: post #11

Not sure why anyone would use 2D CNNs for processing text when there is no spatial correlation in the embedding features. Recent work such as https://arxiv.org/abs/1803.01271 show that for most tasks, 1D CNNs outperform recurrent architectures while being faster to train

This is just a bug in their code. The paper they cite uses 1D convolutions. Though, I suppose having an unused dimension only really hurts efficiency.

> Though, I suppose having an unused dimension only really hurts efficiency.

That might not be true as it might increase bias and thus might need a more careful hyperparameter tuning to avoid overfitting.

Re: What Kagglers Are Using for Text Classification

#56

Post by dna_polymerase »

995533 wrote: post #43
Cybiote wrote: post #34

Earlier quoted context omitted.

> Nobody cares how long it takes to train a model. This isn't true. It depends on your priorities and goals. Machine learning that spends most of its time unable to learn is not real AI. Some of us are interested in sample and energy efficient learning capable of on-line incremental updates immune to catastrophic forgetting. Not just because this is truer to actual learning but because it moves away from being depend…

Nobody in business cares if you are doing proper AI or dumb curve fitting. What matters is the complexity (engineering debt) and performance (accuracy, robustness). Online learning, sample - and energy efficiency are unrelated to training times. Like said: nobody cares if you ran Vowpal Wabbit for 1 hour or 100 hours, as long as you are not constantly babysitting it and calling that paid work (or have the unusual req…

> Nobody in business cares if you are doing proper AI or dumb curve fitting.

What is proper AI? It's all dumb curve fitting right now.

Re: What Kagglers Are Using for Text Classification

#57

Post by Fomite »

995533 wrote: post #51
Fomite wrote: post #49

Earlier quoted context omitted.

No. Resources are not infinite, and we were already on the edge of what the resources at most sites where training would be done could be expected to have.

Thanks. I think you are a correct exception to what I said. I should have known that using words like "nobody" would not go over well on HN (but tedious to type "a very large percentage"), despite that statement being a verbatim quote from one of the world's leading ML engineers and, to me, not controversial. I do consider the cloud both widely available and near infinite in resource adding capability. If it is reall…

So the problem in my circumstance is two-fold:

1) The ML experts in the field have all, pretty much, settled on the need for a uniform method to train models, but for each model needing to be trained on-site.

2) While the cloud might be near infinite in terms of adding capacity, "Hey guys, lets stage up some health-data compatible AWS instances to do something that was a side project we're not even sure will work" in what is always a cash-starved part of healthcare is...well...a pretty big ask.

Re: What Kagglers Are Using for Text Classification

#58

Post by digitalzombie »

995533 wrote: post #23
minimaxir wrote: post #14

It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.

Nobody cares how long it takes to train a model. What matters is prediction speeds, which are comparable (and NLP less likely to require high frequency, where a few more milliseconds matters). Besides that, the accuracy gains are not marginal anymore (BoW can't compete like it used to, especially with pre-trained models).

> Nobody cares how long it takes to train a model.

That's a reckless generalization. I care.

My thesis would take forever if I didn't do any optimization. Also my data is 20 rows with ~6000 predictors.

There are models out there that can take months! I worked on one that took months. We had to tweak it and optimize it to see if we can get it to acceptable training time.

Re: What Kagglers Are Using for Text Classification

#59

Post by richk449 »

995533 wrote: post #51
Fomite wrote: post #49

Earlier quoted context omitted.

No. Resources are not infinite, and we were already on the edge of what the resources at most sites where training would be done could be expected to have.

Thanks. I think you are a correct exception to what I said. I should have known that using words like "nobody" would not go over well on HN (but tedious to type "a very large percentage"), despite that statement being a verbatim quote from one of the world's leading ML engineers and, to me, not controversial. I do consider the cloud both widely available and near infinite in resource adding capability. If it is reall…

> Thanks. I think you are a correct exception to what I said. I should have known that using words like "nobody" would not go over well on HN (but tedious to type "a very large percentage")

In the future, you could use “most”.

Re: What Kagglers Are Using for Text Classification

#60

Post by helpme3 »

elyase wrote: post #22

We have done extensive testing in the context of chatbot intent classification and in our particular problem nothing (including CNN, LSTM, fasttext plus LUIS, Watson and other proprietary classifiers) has been able to beat a simple linear model trained on char n-gram features.

This isn't that surprising. I think the reason for this is that, even though the model is linear, the space of n-grams is so large that there usually is a line that separates any two classes.
Post reply on HN