Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

State-of-the-art text classification with universal language models

nlp.fast.ai

Post reply on HN
Print view View original

31–40 of 54 posts

Re: State-of-the-art text classification with universal language models

#31

Post by ramanan »

jph00 wrote: post #29
ramanan wrote: post #28

I understand that you do mention the pre-training / transfer learning approach clearly, but isn't it disingenuous to claim that you provide better performance based on (only) 100 labeled examples, when the pre-training dataset (Wikitext-103) actually contains 103M words?

Of course not. The use of pre-training on a large unlabeled corpus and subsequent fine-tuning is what the paper is about. It is stated repeatedly in the paper and the post. It is totally correct and in no way misleading to say we need only 100 labeled examples. Anyone can get similar results on their own datasets without even needing to train their own wikitext model, since we've made the pre-trained model available.…

Got it. I was looking for input on how generalizable (the ability of weights to change/adapt) when the training labeled data is 100x smaller than the initial pre-training dataset?

Also, I don't understand the need to be so defensive though and the relevance between my employer and my post?

Re: State-of-the-art text classification with universal language models

#32

Post by Eridrus »

jph00 wrote: post #2

Jeremy here (co-author of this paper). Let me know if you have any questions!

When I think of state of the art in this area, I think of the Deep Contextualized Word Vectors/ELMo paper from Peters et al, which you cite, but you don't have any comparisons to.

The only point of reference between the two papers I see is the CoVe models, which you guys beat pretty handily, but the ELMo model also beats the CoVe model handily, just on different datasets, so not clear how they stack up.

Any chance you could do some more direct comparisons? You do say that it's a more complex architecture, and the tokenized char convolution stuff is a bit of a pain to do, but if that actually helps, it's not that bad to do once.

From an engineering perspective, not changing the LM weights is kind of nice because then you can train multiple separate models on top of the embeddings without needing to retrain everything (and deal with the associated "noise" when retraining models) and it gives some nice modularity. It would be nice to know how much of the performance is lost from having embeddings that can be shared across a lot of tasks.

Random note: it seems like in Table 7, you have bolded "Freez + discr + stlr" in the IMDb column which has a value of 5.00, whereas "Full + discr" has a value of 4.57, and so should probably be the bolded number.

Re: State-of-the-art text classification with universal language models

#33

Post by taneq »

Radim wrote: post #22
jph00 wrote: post #15

Earlier quoted context omitted.

fasttext is just an encoding of the first layer of a model (the word embeddings - or subword embeddings). Full multi-layer pre-trained models are able to do a lot more. For instance, on IMDb sentiment our method is about twice as accurate as fasttext. As to whether CNNs can replace RNNs in general, the jury is still out. Over the last couple of years there have been some sequence tasks where CNNs are state of the art…

For instance, on IMDb sentiment our method is about twice as accurate as fasttext. Seeing as fasttext accuracy is 90%+, does this mean your method achieves 180%? I'm nitpicking of course, but lately I've seen claims like "20% improvement in accuracy", where on closer inspection, the authors mean error rate dropped from 5% to 4%. Which is not bad of course, but in the grand of scheme of things, 1% absolute improvement…

Generally % change in failure rate is what you care about, in most fields. e.g. If something "increases your chance of getting cancer by 50%" that doesn't mean it increases the risk to 1 in 2. It just means the risk goes from 1% to 1.5%

Re: State-of-the-art text classification with universal language models

#34

Post by jph00 »

Eridrus wrote: post #32
jph00 wrote: post #2

Jeremy here (co-author of this paper). Let me know if you have any questions!

When I think of state of the art in this area, I think of the Deep Contextualized Word Vectors/ELMo paper from Peters et al, which you cite, but you don't have any comparisons to. The only point of reference between the two papers I see is the CoVe models, which you guys beat pretty handily, but the ELMo model also beats the CoVe model handily, just on different datasets, so not clear how they stack up. Any chance yo…

Very interesting comments. Yes absolutely want to do comparisons to ELMo. It's a little tricky to do so on our datasets, since ELMo isn't really a complete method on its own, but more an addendum to existing methods. In the future we hope to do seq2seq and sequence labeling studies, and we can then ensure we pick datasets that the ELMo paper covered.

Using char tokens can definitely be helpful, as can sub-words. It's something we've been working on too, and hope to show results of this in the future.

I mainly disagree with your view of end-to-end training. In computer vision we pretty much gave up on trying to re-use hyper-columns without fine-tuning, because the fine-tuning just helps so much. It's really no trouble doing the fine-tuning - in fact the consistency of using a single model form across so many different datasets is really convenient and helpful for doing additional levels of transfer learning.

Thanks for the note about table 7 - it's actually an error (it should be 5.57, not 4.57; Sebastian is in the process of uploading a corrected version).

Re: State-of-the-art text classification with universal language models

#36

Post by jph00 »

Abundnce10 wrote: post #30
jph00 wrote: post #2

Jeremy here (co-author of this paper). Let me know if you have any questions!

This method dramatically improves over previous approaches to text classification, and the code and pre-trained models allow anyone to leverage this new approach to better solve problems such as: Finding documents relevant to a legal case; Identifying spam, bots, and offensive comments; Classifying positive and negative reviews of a product; Grouping articles by political orientation; I'm starting a new project where…

I'm not sure - what you're describing is information extraction. I haven't tried that yet, but I'm certainly interested in doing so (especially for medical data).

Re: State-of-the-art text classification with universal language models

#37

Post by JPKab »

ramanan wrote: post #31
jph00 wrote: post #29

Earlier quoted context omitted.

Of course not. The use of pre-training on a large unlabeled corpus and subsequent fine-tuning is what the paper is about. It is stated repeatedly in the paper and the post. It is totally correct and in no way misleading to say we need only 100 labeled examples. Anyone can get similar results on their own datasets without even needing to train their own wikitext model, since we've made the pre-trained model available.…

Got it. I was looking for input on how generalizable (the ability of weights to change/adapt) when the training labeled data is 100x smaller than the initial pre-training dataset? Also, I don't understand the need to be so defensive though and the relevance between my employer and my post?

When you use the word disingenuous, you invited the response you got. Totally uncalled for to write that.

His response on your employer was likely driven by an assumption that you viewed this as free, open source competition to your product, and thus the negative comment.

To the OP:. I've find a lot of NLP, and this is phenomenal work.

Re: State-of-the-art text classification with universal language models

#38

Post by state_less »

jph00 wrote: post #36
Abundnce10 wrote: post #30

Earlier quoted context omitted.

This method dramatically improves over previous approaches to text classification, and the code and pre-trained models allow anyone to leverage this new approach to better solve problems such as: Finding documents relevant to a legal case; Identifying spam, bots, and offensive comments; Classifying positive and negative reviews of a product; Grouping articles by political orientation; I'm starting a new project where…

I'm not sure - what you're describing is information extraction. I haven't tried that yet, but I'm certainly interested in doing so (especially for medical data).

Similarly, could something like this be useful to extract out a command that a user wants to run from a transcription? For example, "Add a user named Jenny to our client list.", which results in the command, 'create user Jenny'. Or, "Could you add Jenny to our client list?", which results in the same command, 'create user Jenny'. Perhaps instead of outputting the next word, output the expected command from a set of commands?

Re: State-of-the-art text classification with universal language models

#40

Post by timClicks »

jph00 wrote: post #2

Jeremy here (co-author of this paper). Let me know if you have any questions!

Hi Jeremy (& Sebastian of course), this is an amazing contribution to the state of the art!

I haven't read things in depth, but am curious - how do these models cope with out of vocabulary terms?

Post reply on HN