Wow - great paper! Very readable / accessible. I'm working on some stuff for NLP in materials science academic literature, but we haven't tried anything beyond the usual word embedding -> supervised classifier approach. I'll have to give this a try!
State-of-the-art text classification with universal language models
21–30 of 54 posts
Re: State-of-the-art text classification with universal language models
#22I'm glad we're again concentrating on newer language models. Curious how it'll perform compared to fasttext when used as encoding network in larger tasks. I can't help but notice the trend of going back to simpler models with smarter optimizations and regularization to achieve better results. This is a frequent question of mine, which I ask to everyone using RNNs - what do you think of the idea that CNNs will be able…
fasttext is just an encoding of the first layer of a model (the word embeddings - or subword embeddings). Full multi-layer pre-trained models are able to do a lot more. For instance, on IMDb sentiment our method is about twice as accurate as fasttext. As to whether CNNs can replace RNNs in general, the jury is still out. Over the last couple of years there have been some sequence tasks where CNNs are state of the art…
For instance, on IMDb sentiment our method is about twice as accurate as fasttext.
Seeing as fasttext accuracy is 90%+, does this mean your method achieves 180%?
I'm nitpicking of course, but lately I've seen claims like "20% improvement in accuracy", where on closer inspection, the authors mean error rate dropped from 5% to 4%.
Which is not bad of course, but in the grand of scheme of things, 1% absolute improvement may not be such game-changer, especially if it comes at the cost of other relevant metrics like model complexity, developer sanity or performance.
(haven't read your paper yet, just a general sigh/rant)
Re: State-of-the-art text classification with universal language models
#23Jeremy here (co-author of this paper). Let me know if you have any questions!
Hey Jeremy, thanks for sharing this awesome article!
Do you think this is also applicable to classify 1. readability and 2. entertainment/fun of a text? Thanks!
Re: State-of-the-art text classification with universal language models
#24Earlier quoted context omitted.
fasttext is just an encoding of the first layer of a model (the word embeddings - or subword embeddings). Full multi-layer pre-trained models are able to do a lot more. For instance, on IMDb sentiment our method is about twice as accurate as fasttext. As to whether CNNs can replace RNNs in general, the jury is still out. Over the last couple of years there have been some sequence tasks where CNNs are state of the art…
For instance, on IMDb sentiment our method is about twice as accurate as fasttext. Seeing as fasttext accuracy is 90%+, does this mean your method achieves 180%? I'm nitpicking of course, but lately I've seen claims like "20% improvement in accuracy", where on closer inspection, the authors mean error rate dropped from 5% to 4%. Which is not bad of course, but in the grand of scheme of things, 1% absolute improvement…
This generally is the metric you care about - a difference of one percentage point can be an improvement of twenty percent, as that means that the total number of "bad events" that you expect to get when running the system is decreased by 20%. And it's quite reasonable to assume that here, as in almost all other domains, "x% improvement" means the percentage difference (multiplicative), not the percentage point difference (subtractive). For pretty much every percentage quantity, things like defect ratios, recidivism rates or financial interest rates, "20% increase" never means an increase of 20 percentage points but an increase by 20 percent of the starting value. If we're nitpicking, "1% absolute improvement" is an inaccurate statement, the improvement should be described as 1pp (or 20%), not 1%.
Especially for more well defined problems, going from 98.5% to 99.5% is "just" 1pp absolute improvement but the fact that you have three times less mistakes can well justify a more complex model that requires ten times more hardware. The metric that you'd actually care about would often be like "number of hours required to correct the mistakes" or "number of lost sales due to mistakes", which all would get modified by the relative percentage change.
Re: State-of-the-art text classification with universal language models
#25Earlier quoted context omitted.
fasttext is just an encoding of the first layer of a model (the word embeddings - or subword embeddings). Full multi-layer pre-trained models are able to do a lot more. For instance, on IMDb sentiment our method is about twice as accurate as fasttext. As to whether CNNs can replace RNNs in general, the jury is still out. Over the last couple of years there have been some sequence tasks where CNNs are state of the art…
For instance, on IMDb sentiment our method is about twice as accurate as fasttext. Seeing as fasttext accuracy is 90%+, does this mean your method achieves 180%? I'm nitpicking of course, but lately I've seen claims like "20% improvement in accuracy", where on closer inspection, the authors mean error rate dropped from 5% to 4%. Which is not bad of course, but in the grand of scheme of things, 1% absolute improvement…
> but in the grand of scheme of things, 1% absolute improvement may not be such game-changer, especially if it comes at the cost of other relevant metrics like model complexity, developer sanity or performance
fasttext makes errors about 10% of the time, and our approach makes errors about 5% of the time. It's certainly fair to say (although nitpicky) that "accuracy" isn't quite the right term here (I should have said "half the error").
But as for your general sigh/rant... absolute improvement is very rarely the interesting measure. Relative improvement tells you how much your existing systems will change. So if you're error goes from 5% to 4% then you have 20% less errors to deal with than you used to.
An interesting example: the Kaggle Carvana segmentation competition had a lot of competitors complaining that the simple baseline models were so accurate that the competition was pointless (it was very easy to get 99% accuracy). The competition administrator explained however that the purpose of the segmentation model was to do automatic image pasting into new backgrounds, where every mis-classified pixel would lead to image problems (and in a million+ pixels, that's a low error rate!)
Re: State-of-the-art text classification with universal language models
#26Jeremy here (co-author of this paper). Let me know if you have any questions!
Hey Jeremy, thanks for sharing this awesome article! Do you think this is also applicable to classify 1. readability and 2. entertainment/fun of a text? Thanks!
Yes I think it would work great for both of those applications. For readability, you'd ideally want a dataset showing how long someone took to read each document. For entertainment, some kind of review dataset would be fine (it's basically a type of sentiment analysis, which we've already tested on 3 datasets.)
Re: State-of-the-art text classification with universal language models
#27Jeremy here (co-author of this paper). Let me know if you have any questions!
Does the (future) sequence labeling project refers to Part Of Speech tagging?
Re: State-of-the-art text classification with universal language models
#28I understand that you do mention the pre-training / transfer learning approach clearly, but isn't it disingenuous to claim that you provide better performance based on (only) 100 labeled examples, when the pre-training dataset (Wikitext-103) actually contains 103M words?
Re: State-of-the-art text classification with universal language models
#29I understand that you do mention the pre-training / transfer learning approach clearly, but isn't it disingenuous to claim that you provide better performance based on (only) 100 labeled examples, when the pre-training dataset (Wikitext-103) actually contains 103M words?
Of course not. The use of pre-training on a large unlabeled corpus and subsequent fine-tuning is what the paper is about. It is stated repeatedly in the paper and the post.
It is totally correct and in no way misleading to say we need only 100 labeled examples. Anyone can get similar results on their own datasets without even needing to train their own wikitext model, since we've made the pre-trained model available.
(BTW, I see you work at a company that sells something that claims to "categorize SKUs to a standard taxonomy using neural networks." This seems like something you maybe could have mentioned.)
Re: State-of-the-art text classification with universal language models
#30Jeremy here (co-author of this paper). Let me know if you have any questions!
This method dramatically improves over previous approaches to text classification, and the code and pre-trained models allow anyone to leverage this new approach to better solve problems such as: Finding documents relevant to a legal case; Identifying spam, bots, and offensive comments; Classifying positive and negative reviews of a product; Grouping articles by political orientation;
I'm starting a new project where I'm given many recipes and I need to take in a free form text of recipe ingredients (e.g. "1/2 cup diced onions", "two potatoes, cut into 1-inch cubes", etc.) and build a program that identifies the ingredient (e.g. onion, potato), as well as the quantity (e.g. 0.5 cup, 2.0 units). Could I use something like Fast.ai to tackle this problem?