An ImageNet-like text classification task based on Reddit posts
1–10 of 16 posts
Re: An ImageNet-like text classification task based on Reddit posts
#2Fun fact: Reddit uses a similar NLP technique as the t-SNE vizzes to combine the content of subreddits for building recommendations: https://www.youtube.com/watch?v=tKISLQ87GO8
Re: An ImageNet-like text classification task based on Reddit posts
#3This looks fantastic. In particular, the focus on many-class classification is important, it's a common real-world task that is often overlooked. I have some suggestions:
More types of baseline accuracy measure would be useful, eg. accuracy, and micro and macro f1 with unbalanced classes.
It would be very useful to know inter-annotator agreement for the manual classification and human performance for the task of identifying the original subreddit. I'm not a huge fan of creating artificial categories when natural ones are available. In practice there will be a real difference between the 26th and 27th league of legends subreddit, it might be some subtle topical focus shift or something political or tonal.
Is there some kind of standard measure for trading precision and recall when classifying in a hierarchical class structure? That is, you start predicting general high-level categories and move down to the most specific class you can get to before confidence falls below a threshold? Then the evaluation measure gives you more credit for getting lower down the tree (rewarding information gain in the class hierarchy).
Re: An ImageNet-like text classification task based on Reddit posts
#4The post mentions not getting great results with OpenAI Transformer. I haven't tried that, but using a similar framework, ULM-FiT, I narrowly beat the fasttext benchmark on a 250-class dataset we use internally. I will follow up with how it does on this data set.
Re: An ImageNet-like text classification task based on Reddit posts
#5This looks fantastic. In particular, the focus on many-class classification is important, it's a common real-world task that is often overlooked. I have some suggestions: More types of baseline accuracy measure would be useful, eg. accuracy, and micro and macro f1 with unbalanced classes. It would be very useful to know inter-annotator agreement for the manual classification and human performance for the task of iden…
IME, the naive solution to hierarchical classification - building a classifier for each level of the hierarchy - gets me to ~85% accuracy, compared to ~75% accuracy using a "global" classifier.
Re: An ImageNet-like text classification task based on Reddit posts
#6The post mentions not getting great results with OpenAI Transformer. I haven't tried that, but using a similar framework, ULM-FiT, I narrowly beat the fasttext benchmark on a 250-class dataset we use internally. I will follow up with how it does on this data set.
Do any freelance work? We have a small fastText project. Email in profile if you're interested.
Re: An ImageNet-like text classification task based on Reddit posts
#7Fun fact: Reddit uses a similar NLP technique as the t-SNE vizzes to combine the content of subreddits for building recommendations: https://www.youtube.com/watch?v=tKISLQ87GO8
Thanks for posting this. Super cool to see how they are suggesting subreddits
Re: An ImageNet-like text classification task based on Reddit posts
#8The post mentions not getting great results with OpenAI Transformer. I haven't tried that, but using a similar framework, ULM-FiT, I narrowly beat the fasttext benchmark on a 250-class dataset we use internally. I will follow up with how it does on this data set.
ULM-FiT and OpenAI's Transformer* are quite different. Both are pretrained language-models, but ULM-FiT is a standard stack of LSTMs with a particular recipe for fine-tuning, whereas the OpenAI's Transformer uses the much newer Transformer architecture, and no really fancy tricks in the actual fine-tuning. I suspect the difficulty is with the Transformer model itself - this is not the first time I've heard that it is difficult to train.
* = To be clear, this refers to OpenAI's pretrained Transformer model. The Transformer architecture was from work at Google.
Re: An ImageNet-like text classification task based on Reddit posts
#9Are you breaking Reddit TOS by storing/hosting posts?
If someone deletes their post on Reddit it will still be stored and available on your site.
Re: An ImageNet-like text classification task based on Reddit posts
#10Fun fact: Reddit uses a similar NLP technique as the t-SNE vizzes to combine the content of subreddits for building recommendations: https://www.youtube.com/watch?v=tKISLQ87GO8
Great video. A bit sickening to listen to him talk about how great and addicting their site is, but an interesting video on their analytics tech