I'm wondering if these tasks have a form of bias that decreases the performance. If the model sees only positive examples and no negatives then it is biased on the positive paths of decisions. The moment where one changes the path to be incorrect, the model can't recover from the mistake because there weren't any negative examples during pretraining. There's many words that never follow some words but the model never sees that.
Self Supervised Learning in NLP
11–16 of 16 posts
Re: Self Supervised Learning in NLP
#12It’s like one of those low effort medium articles. But not low effort, and not on medium. I really enjoyed it.
That's a weird compliment.
Re: Self Supervised Learning in NLP
#13Hello, number 9) doesn't say what the task is?
Also, I always wondered, do those methods work universally on all languages? For example Chinese, Korean and Japanese, with different alphabets.
Re: Self Supervised Learning in NLP
#14Re: Self Supervised Learning in NLP
#15Earlier quoted context omitted.
Even humans fails on this very often.
Haha, just imagine if an AI faces Chandler
Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)
Re: Self Supervised Learning in NLP
#16Hello, number 9) doesn't say what the task is? Also, I always wondered, do those methods work universally on all languages? For example Chinese, Korean and Japanese, with different alphabets.
I've elaborated number 9. Please let me know if it's clear now.
Some of these methods do. For example, the tasks that power word vectors apply for many languages. https://fasttext.cc/docs/en/crawl-vectors.html
Masked Language Modeling has been applied to learn cross-lingual language models. Look into: https://arxiv.org/abs/1901.07291