Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Kagglers Are Using for Text Classification

mlwhiz.com

Post reply on HN
Print view View original

61–70 of 72 posts

Re: What Kagglers Are Using for Text Classification

#61

Post by 995533 »

dna_polymerase wrote: post #56
995533 wrote: post #43

Earlier quoted context omitted.

Nobody in business cares if you are doing proper AI or dumb curve fitting. What matters is the complexity (engineering debt) and performance (accuracy, robustness). Online learning, sample - and energy efficiency are unrelated to training times. Like said: nobody cares if you ran Vowpal Wabbit for 1 hour or 100 hours, as long as you are not constantly babysitting it and calling that paid work (or have the unusual req…

> Nobody in business cares if you are doing proper AI or dumb curve fitting. What is proper AI? It's all dumb curve fitting right now.

Oh, baby don't hurt me. Don't hurt me. No more.

Re: What Kagglers Are Using for Text Classification

#62

Post by autokad »

minimaxir wrote: post #14

It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.

an active kaggler here.

I dont use NN because they simply don't have great accuracy, and most importantly they have a huge amount of variance. this is mostly because the data on kaggle is not very large. the gbm trifecta (xgboost, catboost, lgbm) also does really really well.

> "Kaggle prioritizes chasing a metric, but real-world data science has more considerations."

this is counter to your point. most real-world considerations need things like model explanability.

I notice that things dont make hacker news if they are doing anything other than NN. model with xgboost gets X% accuracy - crickets. model with X-Y% accuracy with DNN - headline news.

I also notice that teams in industry tend to throw a DNN at a problem and never try something more simpler like xgboost. I saw a team with an LSTM for text lament they had 80% accuracy on training/evaluation, but when pushed to prod dropped down to 50%. I saw the errors they were getting, I said: maybe its too complicated and not generalizing well, have you tried xgboost?

they retorted LSTM's w/ word2vec is very robust. I thought, obviously its not given your results. I tried to offer the idea that word2vec was trained on an entirely different kind of corpus (also a problem when trying to use word2vec in kaggle)

Re: What Kagglers Are Using for Text Classification

#63

Post by autokad »

995533 wrote: post #23
minimaxir wrote: post #14

It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.

Nobody cares how long it takes to train a model. What matters is prediction speeds, which are comparable (and NLP less likely to require high frequency, where a few more milliseconds matters). Besides that, the accuracy gains are not marginal anymore (BoW can't compete like it used to, especially with pre-trained models).

> "Nobody cares how long it takes to train a model."

In kaggle some competitions it takes over 7 hours to train a model, and I can generally think of 10 things a day to try. prediction only takes about a minute.

> "especially with pre-trained models" if the corpus are different, pre-trained models do not help much, if not hurt.

Re: What Kagglers Are Using for Text Classification

#64

Post by sandeepeecs »

rundigen12 wrote: post #37
995533 wrote: post #23

Earlier quoted context omitted.

Nobody cares how long it takes to train a model. What matters is prediction speeds, which are comparable (and NLP less likely to require high frequency, where a few more milliseconds matters). Besides that, the accuracy gains are not marginal anymore (BoW can't compete like it used to, especially with pre-trained models).

> Nobody cares how long it takes to train a model. LOTS of people care how long it takes to train a model. A few minutes, vs. a day, vs. a week, vs. a month? Yea, that matters. Think about how long it takes to try out different hyperparameters or make other adjustments while conducting research... If you're Google maybe you don't care as much because you can fire off a hundred different jobs at once, but if you're a…

Yes I agree. most people who come to us at alpes AI do care about training time. how fast they can do experiments

Another important aspect is training and incremental training on edge device.

At the time when privacy is becoming very important and you cannot export data from mobile devices etc. Training time on mobile is an important factor

Re: What Kagglers Are Using for Text Classification

#65

Post by scottlocklin »

soraki_soladead wrote: post #21
minimaxir wrote: post #14

It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.

For most businesses a GPU is not really expensive, particularly if the results are important to the business. Chasing a single metric is certainly too narrow but getting the best results often does matter in a professional context too. While Kaggle can go overboard on massive ensembles using a state-of-the-art approach to the problem is often warranted outside of Kaggle.

Do you have an example backing up your assertions?

In my experience, the worst text classification is usually fine. Labels are usually too inaccurate and subjective for "accuracy" to matter much.

Re: What Kagglers Are Using for Text Classification

#66

Post by soraki_soladead »

scottlocklin wrote: post #65
soraki_soladead wrote: post #21

Earlier quoted context omitted.

For most businesses a GPU is not really expensive, particularly if the results are important to the business. Chasing a single metric is certainly too narrow but getting the best results often does matter in a professional context too. While Kaggle can go overboard on massive ensembles using a state-of-the-art approach to the problem is often warranted outside of Kaggle.

Do you have an example backing up your assertions? In my experience, the worst text classification is usually fine. Labels are usually too inaccurate and subjective for "accuracy" to matter much.

Electronic medical records. Anything in healthcare/medical, financial services, legal. There's a long way from the "worst text classification" to "sufficient text classification" in most real world use cases. With a reasonable budget you can relabel data and work around subjectivity.

Re: What Kagglers Are Using for Text Classification

#67

Post by soraki_soladead »

autokad wrote: post #62
minimaxir wrote: post #14

It should be noted that CNNs and LSTMs are an order of magnitude slower than things like bag-of-words/fasttext unless you're using an expensive GPU, and the accuracy benefit if any may be marginal in practice. Kaggle prioritizes chasing a metric, but real-world data science has more considerations.

an active kaggler here. I dont use NN because they simply don't have great accuracy, and most importantly they have a huge amount of variance. this is mostly because the data on kaggle is not very large. the gbm trifecta (xgboost, catboost, lgbm) also does really really well. > "Kaggle prioritizes chasing a metric, but real-world data science has more considerations." this is counter to your point. most real-world co…

I agree good models like xgboost get buried and inexperienced practitioners jump to deep models too quickly. Often without understanding how to properly architect and tune them. Always start with a simple baseline (EDIT: and good process).

However, what's lacking in the ML practitioner community is nuance. Some applications need deep models some problems need xgboost. There isn't a "best" model in text classification because it depends on your data and problem.

Re: What Kagglers Are Using for Text Classification

#68

Post by SpaceManNabs »

Seems like the article is mistaken on one part.

Attention was first coined in this paper (as far as I know): https://arxiv.org/pdf/1409.0473v7.pdf

The second page of the introduction of the "Hierarchical Attention Networks for Document Classification" paper mentioned in the article even cites it.

Re: What Kagglers Are Using for Text Classification

#69

Post by asah »

soraki_soladead wrote: post #67
autokad wrote: post #62

Earlier quoted context omitted.

an active kaggler here. I dont use NN because they simply don't have great accuracy, and most importantly they have a huge amount of variance. this is mostly because the data on kaggle is not very large. the gbm trifecta (xgboost, catboost, lgbm) also does really really well. > "Kaggle prioritizes chasing a metric, but real-world data science has more considerations." this is counter to your point. most real-world co…

I agree good models like xgboost get buried and inexperienced practitioners jump to deep models too quickly. Often without understanding how to properly architect and tune them. Always start with a simple baseline (EDIT: and good process). However, what's lacking in the ML practitioner community is nuance. Some applications need deep models some problems need xgboost. There isn't a "best" model in text classification…

This chart is great for newbies, perhaps give it an update?

https://scikit-learn.org/stable/tutorial/machine_learning_ma...

Re: What Kagglers Are Using for Text Classification

#70

Post by ykevinator »

rundigen12 wrote: post #32

I was really hoping to see a summary comparison of the performance(s) of the different models at the end, e.g. accuracy vs. complexity vs. execution time, etc. Here's a summary from the end of each section... 1. TextCNN: "This kernel scored around 0.661 on the public leaderboard." 2. BiDirectional RNN: 0.671 3. Attention Models: 0.682

Thank you, that's exactly what I was scrolling through the bickering to find.
Post reply on HN