51–60 of 69 posts
I think the big reason why BERT and T5 have fallen out of favor is the lack of zero shot (or few shot) ability. When you have hundreds or thousands of examples, BERT works great. But that is very restricting.
For people like me who gave up trying to follow Arxiv ML papers 3+ years ago, articles like these are gold. I would love a Youtube channel or blog which does retrospectives on "big" papers of the last decade (those that everyone paid attention to at the time) and look at where the ideas are today.
feels like large language models sucked all the air out of the room because it was a lot easier to scale compute and data, and after roberta, no one was willing to continue exploring.
No, there are mathematical reasons LLMs are better. They are trained with multiobjective loss (coding skills, translation skills, etc) so they understand the world much better than MLM. Original post discuss that but with more words and points than necessary.
Earlier quoted context omitted.
Yes but you can use an llm to label data and then train a bert model which then costs a small fraction of time and money to run than the original llm.
Shhh, don’t tell everybody the secret. ;-)
BERT didn’t go anywhere and I have seen fine-tuned BERT backbones everywhere. They are useful for generating embeddings to be used downstream, and small enough to be handled on consumer (pre Ampere) hardware. One of the trends I have seen is scaling BERT down rather than up, since BERT already gave good performance, we want to be able to do it faster and cheaper. That gave rise to RoBERTa, ALBERT and distillBERT. T5…
I tried some large scale translation tasks with T5 and results were iffy at best. I’m going to try the same task with the newest Mistral small models and compare. My guess is Mistral will be better.
Earlier quoted context omitted.
For text classification/clustering/retrieval I am pretty happy with BERT-family models. It's only the last few month that I've seen better models come out that are practical (e.g. not sell all your children to Open AI to afford them)
Classification is just too damn convenient with LLMs.
If you have labelled training data (or semi-labelled), BERT takes the cake, both in terms of accuracy and efficiency. In fact, you can have luck with getting a CLM to generate noisy labels and then training BERT/RoBERTa on that to get a robust strong classifier.
BERT didn’t go anywhere and I have seen fine-tuned BERT backbones everywhere. They are useful for generating embeddings to be used downstream, and small enough to be handled on consumer (pre Ampere) hardware. One of the trends I have seen is scaling BERT down rather than up, since BERT already gave good performance, we want to be able to do it faster and cheaper. That gave rise to RoBERTa, ALBERT and distillBERT. T5…
https://stability.ai/news/stable-diffusion-3-research-paper
Related discussion
https://www.reddit.com/r/StableDiffusion/comments/1c0by2y/wh...
Earlier quoted context omitted.
Yes but you can use an llm to label data and then train a bert model which then costs a small fraction of time and money to run than the original llm.
Is the encoder style arch better for representing classification tasks at a given compute budget than a causal LM? Is this because the final represention in bert style models more globally focused, rather than being optimized for next token prediction?
Luckily, it is now trivial to drop the post into Claude and say "Re-write this without "
So, just in case you also felt like you were driving over a road filled with potholes trying to read this post, don't just click away, have your handy LLM take a pass at it. There's good stuff to be found.
Yi is a good source in this area, and a good follow on Twitter.