Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

State-of-the-art text classification with universal language models

nlp.fast.ai

Post reply on HN
Print view View original

41–50 of 54 posts

Re: State-of-the-art text classification with universal language models

#41

Post by timClicks »

Abundnce10 wrote: post #30
jph00 wrote: post #2

Jeremy here (co-author of this paper). Let me know if you have any questions!

This method dramatically improves over previous approaches to text classification, and the code and pre-trained models allow anyone to leverage this new approach to better solve problems such as: Finding documents relevant to a legal case; Identifying spam, bots, and offensive comments; Classifying positive and negative reviews of a product; Grouping articles by political orientation; I'm starting a new project where…

If you have sufficient labelled data, conditional random fields works well for this kind of problem. A technical team from NY Times have a great piece on it https://open.blogs.nytimes.com/2015/04/09/extracting-structu...

Re: State-of-the-art text classification with universal language models

#42

Post by gargarplex »

jph00 wrote: post #20
beders wrote: post #12

I've yet to see data this is beating a SVM on a 4-character shingle approach (which also doesn't require tons of data to train).

It beats it easily. See the paper and citations for comparison

Is there an easily accessible API? And is this robust to bad labels - imperfect training data? I have a huge corpus of labeled descriptions for jobs and I want to categorize them as 'programming' or 'not programming'. The accuracy of my manual labeling is like 95%. Can that be used to train a classifier using this newly published technology?

Re: State-of-the-art text classification with universal language models

#43

Post by madisonmay »

jph00 wrote: post #34
Eridrus wrote: post #32

Earlier quoted context omitted.

When I think of state of the art in this area, I think of the Deep Contextualized Word Vectors/ELMo paper from Peters et al, which you cite, but you don't have any comparisons to. The only point of reference between the two papers I see is the CoVe models, which you guys beat pretty handily, but the ELMo model also beats the CoVe model handily, just on different datasets, so not clear how they stack up. Any chance yo…

Very interesting comments. Yes absolutely want to do comparisons to ELMo. It's a little tricky to do so on our datasets, since ELMo isn't really a complete method on its own, but more an addendum to existing methods. In the future we hope to do seq2seq and sequence labeling studies, and we can then ensure we pick datasets that the ELMo paper covered. Using char tokens can definitely be helpful, as can sub-words. It's…

Yes absolutely want to do comparisons to ELMo.

Perhaps even more interesting than comparison would be modifications to ULMFit to incorporate good ideas from the AllenNLP ELMo paper.

The learned weighting of representation layers seems like a decent candidate, as does giving the model flexibility to use something other than a concatenated [mean / max / last state] representation of final LSTM output layer (as is the case in some of ELMo's task models). I'm personally curious about using an attention mechanism in conjunction with something like ELMo's gamma task parameter (regularizer) for learning a weighted combination of outputs but haven't been able to get things to function well in practice.

The dataset the ELMo model is trained might also be preferable to WIKI 103 for practical English tasks, although you lose the nice multilingual benefits you get from working with WIKI 103.

In general it seems like the format described in the ELMo paper is simply not designed to work at very low N because the weights of the (often complex) task models used in ELMo's benchmarks are learned entirely for scratch. That's not possible without a decent amount of labeled training data.

Anyhow, thought the paper was very well put together, definitely an enjoyable read. Hope yourself and Sebastian collaborate on future papers, as good things certainly came of this one!

Re: State-of-the-art text classification with universal language models

#44

Post by Radim »

PeterisP wrote: post #24
Radim wrote: post #22

Earlier quoted context omitted.

For instance, on IMDb sentiment our method is about twice as accurate as fasttext. Seeing as fasttext accuracy is 90%+, does this mean your method achieves 180%? I'm nitpicking of course, but lately I've seen claims like "20% improvement in accuracy", where on closer inspection, the authors mean error rate dropped from 5% to 4%. Which is not bad of course, but in the grand of scheme of things, 1% absolute improvement…

This generally is the metric you care about - a difference of one percentage point can be an improvement of twenty percent, as that means that the total number of "bad events" that you expect to get when running the system is decreased by 20%. And it's quite reasonable to assume that here, as in almost all other domains, "x% improvement" means the percentage difference (multiplicative), not the percentage point diffe…

Yes, that's what I was getting at.

Your note on "more well defined problems" is spot on. Chasing single percent improvements and SOTA is indeed the name of the game there.

But defining the problem in the first place, figuring out the cost matrix and solution constraints, is typically the bigger challenge in highly innovative projects. Once you know what to chase, 80% of the job is done.

Disclosure: building commercial ML systems for the past 11 years, using deep learning and otherwise. What you call "metric you care about" is often not the metric you care about. This is why people coming from academia are sometimes taken by surprise that logistic regression, linear models, or heck, even rule-based systems (!) are still so popular. Model simplicity, developer sanity and performance do matter, too.

Re: State-of-the-art text classification with universal language models

#46

Post by antirez »

pixelHD wrote: post #14

I'm glad we're again concentrating on newer language models. Curious how it'll perform compared to fasttext when used as encoding network in larger tasks. I can't help but notice the trend of going back to simpler models with smarter optimizations and regularization to achieve better results. This is a frequent question of mine, which I ask to everyone using RNNs - what do you think of the idea that CNNs will be able…

Relevant to RNNs vs CNNs -> https://openreview.net/pdf?id=rk8wKk-R-

Re: State-of-the-art text classification with universal language models

#47

Post by Eridrus »

jph00 wrote: post #34
Eridrus wrote: post #32

Earlier quoted context omitted.

When I think of state of the art in this area, I think of the Deep Contextualized Word Vectors/ELMo paper from Peters et al, which you cite, but you don't have any comparisons to. The only point of reference between the two papers I see is the CoVe models, which you guys beat pretty handily, but the ELMo model also beats the CoVe model handily, just on different datasets, so not clear how they stack up. Any chance yo…

Very interesting comments. Yes absolutely want to do comparisons to ELMo. It's a little tricky to do so on our datasets, since ELMo isn't really a complete method on its own, but more an addendum to existing methods. In the future we hope to do seq2seq and sequence labeling studies, and we can then ensure we pick datasets that the ELMo paper covered. Using char tokens can definitely be helpful, as can sub-words. It's…

I just wanted to clear up my comments on fine tuning. These LMs are huge. The ELMo paper has 300 dimensional embeddings, yours has 400 (which, btw, should probably be controlled in a comparison). As an engineer, I don't really want to deploy a fine tuned LM for every task I have. Especially on smartphones, I can barely deploy one of these.

The obvious answer is that I should just train a single joint model.

That's great, but when you retrain a model, even if you get similar accuracy, your actual predictions change. It's basically why same model ensembles help.

So if I am trying to improve predictions for a single task, but I have a joint model, then I have to deal with a whole pile of churn that I wouldn't if I had separate models.

This doesn't show up in academic metrics, but people care when things that used to work stop working for no real reason, even if an equal amount of new things started working.

So, I'm not saying we shouldn't fine tune things, it's that I have a set of engineering challenges that make fine tuning less ideal, and I am curious how much we can get away with sharing. There are plenty of CV papers which indicate that the very first layers basically don't benefit from fine tuning because they are so general. Is that true for NLP as well, or are words embeddings already quite domain specific?

Re: State-of-the-art text classification with universal language models

#48

Post by cs702 »

Radim wrote: post #22
jph00 wrote: post #15

Earlier quoted context omitted.

fasttext is just an encoding of the first layer of a model (the word embeddings - or subword embeddings). Full multi-layer pre-trained models are able to do a lot more. For instance, on IMDb sentiment our method is about twice as accurate as fasttext. As to whether CNNs can replace RNNs in general, the jury is still out. Over the last couple of years there have been some sequence tasks where CNNs are state of the art…

For instance, on IMDb sentiment our method is about twice as accurate as fasttext. Seeing as fasttext accuracy is 90%+, does this mean your method achieves 180%? I'm nitpicking of course, but lately I've seen claims like "20% improvement in accuracy", where on closer inspection, the authors mean error rate dropped from 5% to 4%. Which is not bad of course, but in the grand of scheme of things, 1% absolute improvement…

Radim: please consider incorporating this into gensim. It really is superior to simpler classification models running on top of word/BPE/wordpiece embeddings and to classic machine learning algorithms used for text classification and topic modeling like HDP, LDA, LSI/LSA, etc. (You can see for yourself how well this works out-of-the-box with a simple exercise: grab a pretrained model from fast.ai, run a bunch of documents through it, grabbing and saving each time the last hidden-layer representation of each document, and then map these representations to a two-dimensional plot with, say, t-SNE.)

I realize that outside of Silicon Valley and other technology centers, most established companies are far -- far -- from adopting deep learning for any application of importance, due partly to the current unavailability of developers with AI expertise, and partly to deep learning's so-called "unexplainability" (i.e., the inability of many corporate executives and machine learning practitioners to reason about it, and their resulting discomfort with it). But it's only a matter of time before Corporate America starts following the lead of companies like Google and Facebook, which today are aggressively using state-of-the-art AI in lots of important applications.

Why not get ahead of this multi-decade trend?

PS. For those who don't know, Radim is the creator of gensim, a popular, friendly Python library for text classification and topic modeling.[a]

[a] https://radimrehurek.com/gensim | https://github.com/RaRe-Technologies/gensim

Re: State-of-the-art text classification with universal language models

#49

Post by madisonmay »

timClicks wrote: post #40
jph00 wrote: post #2

Jeremy here (co-author of this paper). Let me know if you have any questions!

Hi Jeremy (& Sebastian of course), this is an amazing contribution to the state of the art! I haven't read things in depth, but am curious - how do these models cope with out of vocabulary terms?

Initially embeddings of OOV terms are initialized to mean embeddings, and then the language model fine-tuning step allows the model to adjust these values.

See [18] here: https://github.com/fastai/fastai/blob/master/courses/dl2/imd....

Re: State-of-the-art text classification with universal language models

#50

Post by ioot17 »

Abundnce10 wrote: post #30
jph00 wrote: post #2

Jeremy here (co-author of this paper). Let me know if you have any questions!

This method dramatically improves over previous approaches to text classification, and the code and pre-trained models allow anyone to leverage this new approach to better solve problems such as: Finding documents relevant to a legal case; Identifying spam, bots, and offensive comments; Classifying positive and negative reviews of a product; Grouping articles by political orientation; I'm starting a new project where…

CRF works quite well, it's actually what I utilize right now to approach recipe parsing on https://cookalo.com/. It's based on CRFsuite with Python bindings for data training on already labeled recipes. If you build your own app and want to do some comparison, feel free to run some benchmarks against it.
Post reply on HN