What happened to BERT and T5?
yitay.net
What happened to BERT and T5?
1–10 of 69 posts
Re: What happened to BERT and T5?
#2Yi is a good source in this area, and a good follow on Twitter.
Re: What happened to BERT and T5?
#3>If BERT worked so well, why not scale it?
I mean, the scaling already happened in 2019 with RoBERTa, my guess is that these models are already good enough at what they need to do (creating meaningful text embeddings), and making them extremely large wasn't feasible for deployment.
Re: What happened to BERT and T5?
#4What happened is that "transformers go whrrrrrr." (yes, that's the academic term)
In the end, LLMs using causal language modeling or masked language modeling learn to best solve their objectives by creating an efficient global model of language patterns: CLM is actually a harder problem to solve since MLM can leak information through surrounding context, and with transformer scaling law research post-BERT/GPT it's not a surprise CLM won out in the long run.
Re: What happened to BERT and T5?
#5>If BERT worked so well, why not scale it? I mean, the scaling already happened in 2019 with RoBERTa, my guess is that these models are already good enough at what they need to do (creating meaningful text embeddings), and making them extremely large wasn't feasible for deployment.
For text classification/clustering/retrieval I am pretty happy with BERT-family models. It's only the last few month that I've seen better models come out that are practical (e.g. not sell all your children to Open AI to afford them)
Re: What happened to BERT and T5?
#6>If BERT worked so well, why not scale it? I mean, the scaling already happened in 2019 with RoBERTa, my guess is that these models are already good enough at what they need to do (creating meaningful text embeddings), and making them extremely large wasn't feasible for deployment.
For text classification/clustering/retrieval I am pretty happy with BERT-family models. It's only the last few month that I've seen better models come out that are practical (e.g. not sell all your children to Open AI to afford them)
What would you say are the better models nowadays that are practical?
Re: What happened to BERT and T5?
#7I think the big reason why BERT and T5 have fallen out of favor is the lack of zero shot (or few shot) ability.
When you have hundreds or thousands of examples, BERT works great. But that is very restricting.
Re: What happened to BERT and T5?
#8>If BERT worked so well, why not scale it? I mean, the scaling already happened in 2019 with RoBERTa, my guess is that these models are already good enough at what they need to do (creating meaningful text embeddings), and making them extremely large wasn't feasible for deployment.
For text classification/clustering/retrieval I am pretty happy with BERT-family models. It's only the last few month that I've seen better models come out that are practical (e.g. not sell all your children to Open AI to afford them)
Classification is just too damn convenient with LLMs.
Re: What happened to BERT and T5?
#9I'm a bit embarrassed to admit, but I still don't understand decoder vs encoder vs decoder/encoder models.
Is the input/output of these models any different? Are they all just "text context goes in, scores for all tokens in the vocabulary come out" ? Is the difference only in how they achieve this output?
Re: What happened to BERT and T5?
#10I believe many high-quality embedding models are still based on BERT, even recent ones, so I don't think it's entirely fair to characterize it as "deprecated".