feels like large language models sucked all the air out of the room because it was a lot easier to scale compute and data, and after roberta, no one was willing to continue exploring.
What happened to BERT and T5?
11–20 of 69 posts
Re: What happened to BERT and T5?
#12I'm a bit embarrassed to admit, but I still don't understand decoder vs encoder vs decoder/encoder models. Is the input/output of these models any different? Are they all just "text context goes in, scores for all tokens in the vocabulary come out" ? Is the difference only in how they achieve this output?
The key to understanding the difference is that transformers are attention models where tokens can "attend" to different tokens.
Encoder models allow all tokens to attend to every other token. This increases the number of connections and makes it easier for the model to reason, but requires all tokens at once to produce any output. These models generally can't generate text.
Decoder models only allow tokens to attend to previous tokens in the sequence. This decreases the amount of tokens, but allows the model to be run incrementally, one token at a time. This incremental processing is key to allowing the models to generate text.
Re: What happened to BERT and T5?
#13I'm a bit embarrassed to admit, but I still don't understand decoder vs encoder vs decoder/encoder models. Is the input/output of these models any different? Are they all just "text context goes in, scores for all tokens in the vocabulary come out" ? Is the difference only in how they achieve this output?
The biggest difference is when you feed a sequence into a decoder only model, it will only attend to previous tokens when computing hidden states for the current token. So the hidden states for the nth token is only based on tokens Encoder architectures have been used for semantic analysis, and feature extraction of sequences, and encoder only for generation (i.e. next token prediction).
Re: What happened to BERT and T5?
#14Earlier quoted context omitted.
For text classification/clustering/retrieval I am pretty happy with BERT-family models. It's only the last few month that I've seen better models come out that are practical (e.g. not sell all your children to Open AI to afford them)
What would you say are the better models nowadays that are practical?
For my recommender/object sorter I have not been in a hurry to upgrade because I have other things to think about. This table should give you some idea of the time-space-accuracy trade offs
https://huggingface.co/spaces/mteb/leaderboard
In a lot of cases you will see two models with a huge difference in size but a tiny difference in accuracy. I could fit either the big or small Stella on my 4080.
Re: What happened to BERT and T5?
#15I'm a bit embarrassed to admit, but I still don't understand decoder vs encoder vs decoder/encoder models. Is the input/output of these models any different? Are they all just "text context goes in, scores for all tokens in the vocabulary come out" ? Is the difference only in how they achieve this output?
You can think of encoder/decoder models as specifically addressing the translation problem. They are also known as sequence-to-sequence models.
Take the task of translation. A translator needs to keep in mind the original text and the translation so far in order to predict the next translated token. The original text is encoded, and the translation so far is passed into the decoder to generate the next translated token. The next token is appended to the translation and the process repeats autoregressively.
Decoder-only models use just the decoder architecture of encoder/decoders. They are prompted and generate completions autoregressively.
Encoder-only models use just the encoder architecture which you can think of similarly to embedding. A task here is, producing vectors where vector distance is related to the semantic similarity of the input documents. This can be useful for retrieval tasks among other things.
You can of course translate using just the decoder, by constructing a "please translate this from A to B, " prompt and generating tokens just using the decoder. I'll leave it to people with more expertise than I do describe the pros and cons of these.
Re: What happened to BERT and T5?
#16Wasn’t there a recent paper that demonstrated BERT models are still competitive or beat LLMs in many tasks?
Re: What happened to BERT and T5?
#17feels like large language models sucked all the air out of the room because it was a lot easier to scale compute and data, and after roberta, no one was willing to continue exploring.
No, there are mathematical reasons LLMs are better. They are trained with multiobjective loss (coding skills, translation skills, etc) so they understand the world much better than MLM. Original post discuss that but with more words and points than necessary.
Re: What happened to BERT and T5?
#18For people like me who gave up trying to follow Arxiv ML papers 3+ years ago, articles like these are gold. I would love a Youtube channel or blog which does retrospectives on "big" papers of the last decade (those that everyone paid attention to at the time) and look at where the ideas are today.
Re: What happened to BERT and T5?
#19I'm a bit embarrassed to admit, but I still don't understand decoder vs encoder vs decoder/encoder models. Is the input/output of these models any different? Are they all just "text context goes in, scores for all tokens in the vocabulary come out" ? Is the difference only in how they achieve this output?
Encoder: Text tokens -> Fixed representation vector
Decoder: Fixed representation vector + N decoded text tokens -> N+1th text token
Encoder/Decoder architecture: You take some tokenized text, run an encoder on it to get a fixed representation vector, and then recursively apply the decoder to your fixed representation vector and the 0...N tokens you've already produced to produce the N+1th token.
Decoder-only architecture: You take some tokenized text, and recursively apply a decoder to the 0...N tokens you've already produced to produce the N+1th token (without ever using an encoded representation vector).
Basically, an encoder produces this intermediate output which a decoder knows how to combine with some existing output to create more output (imagine, e.g., encoding a sentence in French, and then feeding a decoder the vector representation of that sentence plus the three words you've translated so far, so that it can figure out the next word in the translation). A decoder can be made to require an intermediate context vector, or (this is how it's done in decoder-only architectures) it can be made to require only the text produced so far.
Re: What happened to BERT and T5?
#20I'm a bit embarrassed to admit, but I still don't understand decoder vs encoder vs decoder/encoder models. Is the input/output of these models any different? Are they all just "text context goes in, scores for all tokens in the vocabulary come out" ? Is the difference only in how they achieve this output?
Don't be embarrassed. This article makes the mistake of _saying_ they're going catch the under-informed up to speed but then immediately dives all the way in to the deep end.