Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happened to BERT and T5?

yitay.net

Post reply on HN
Print view View original

21–30 of 69 posts

Re: What happened to BERT and T5?

#21

Post by Der_Einzige »

nshm wrote: post #17
htrp wrote: post #11

feels like large language models sucked all the air out of the room because it was a lot easier to scale compute and data, and after roberta, no one was willing to continue exploring.

No, there are mathematical reasons LLMs are better. They are trained with multiobjective loss (coding skills, translation skills, etc) so they understand the world much better than MLM. Original post discuss that but with more words and points than necessary.

Call it a CLM vs MLM, not LLM vs MLM. Soon LMLM's will exist, which will be LLMs too...

Re: What happened to BERT and T5?

#22

Post by opprobium »

ambrozk wrote: post #19
andy_xor_andrew wrote: post #9

I'm a bit embarrassed to admit, but I still don't understand decoder vs encoder vs decoder/encoder models. Is the input/output of these models any different? Are they all just "text context goes in, scores for all tokens in the vocabulary come out" ? Is the difference only in how they achieve this output?

Encoder: Text tokens -> Fixed representation vector Decoder: Fixed representation vector + N decoded text tokens -> N+1th text token Encoder/Decoder architecture: You take some tokenized text, run an encoder on it to get a fixed representation vector, and then recursively apply the decoder to your fixed representation vector and the 0...N tokens you've already produced to produce the N+1th token. Decoder-only archite…

Encoder in the T5 sense doesn't produce a fixed vector, it produces one encoded vector for every step of input and all of that is given to the decoder.

The only difference between encoder/decoder and decoder-only is masking:

In an encoder, none of the tokens are masked at any step, and are all visible in both directions to the encoder. Each output of the encoder can attend to any input of the encoder.

In the decoder, the tokens are masked causally - each N+1 token can only attend to the previous N tokens.

Re: What happened to BERT and T5?

#23

Post by jerrygenser »

lalaland1125 wrote: post #7

I think the big reason why BERT and T5 have fallen out of favor is the lack of zero shot (or few shot) ability. When you have hundreds or thousands of examples, BERT works great. But that is very restricting.

Yes but you can use an llm to label data and then train a bert model which then costs a small fraction of time and money to run than the original llm.

Re: What happened to BERT and T5?

#24

Post by janalsncm »

BERT didn’t go anywhere and I have seen fine-tuned BERT backbones everywhere. They are useful for generating embeddings to be used downstream, and small enough to be handled on consumer (pre Ampere) hardware. One of the trends I have seen is scaling BERT down rather than up, since BERT already gave good performance, we want to be able to do it faster and cheaper. That gave rise to RoBERTa, ALBERT and distillBERT.

T5 I have worked less with but I would be curious about its head to head performance with decoder-only models these days. My guess is the downsides from before (context window limitations) are less of a factor than they used to be.

Re: What happened to BERT and T5?

#25

Post by byefruit »

lalaland1125 wrote: post #7

I think the big reason why BERT and T5 have fallen out of favor is the lack of zero shot (or few shot) ability. When you have hundreds or thousands of examples, BERT works great. But that is very restricting.

Yes, no zero shot. Few shot is possible for some use cases with setfit: https://github.com/huggingface/setfit and the very recent Fastfit: https://github.com/IBM/fastfit ( https://arxiv.org/pdf/2404.12365 )

Re: What happened to BERT and T5?

#27

Post by thomasahle »

andy_xor_andrew wrote: post #9

I'm a bit embarrassed to admit, but I still don't understand decoder vs encoder vs decoder/encoder models. Is the input/output of these models any different? Are they all just "text context goes in, scores for all tokens in the vocabulary come out" ? Is the difference only in how they achieve this output?

If you look at the classical [transformer architecture picture](https://en.wikipedia.org/wiki/Transformer_(deep_learning_arc...) there is an "encoder" tower on the left and a "decoder" tower on the right.

- Bert is encoder only.

- GPT is decoder only.

- T5 uses both the encoder and the decoder.

Re: What happened to BERT and T5?

#28

Post by hdhshdhshdjd »

Maybe in SOTA ml/nlp research, but in the world of building useful tools and products, BERT models are dead simple to tune, work great if you have decent training data, and most importantly are very very fast and very very cheap to run.

I have a small Swiss army collection of custom BERT fine tunes that are equal or better than the best LLM and execute document classification tasks in 2.4ms. Find me an LLM that can do anything in 2.4ms.

Re: What happened to BERT and T5?

#29

Post by hdhshdhshdjd »

janalsncm wrote: post #24

BERT didn’t go anywhere and I have seen fine-tuned BERT backbones everywhere. They are useful for generating embeddings to be used downstream, and small enough to be handled on consumer (pre Ampere) hardware. One of the trends I have seen is scaling BERT down rather than up, since BERT already gave good performance, we want to be able to do it faster and cheaper. That gave rise to RoBERTa, ALBERT and distillBERT. T5…

I tried some large scale translation tasks with T5 and results were iffy at best. I’m going to try the same task with the newest Mistral small models and compare. My guess is Mistral will be better.

Re: What happened to BERT and T5?

#30

Post by hdhshdhshdjd »

jerrygenser wrote: post #23
lalaland1125 wrote: post #7

I think the big reason why BERT and T5 have fallen out of favor is the lack of zero shot (or few shot) ability. When you have hundreds or thousands of examples, BERT works great. But that is very restricting.

Yes but you can use an llm to label data and then train a bert model which then costs a small fraction of time and money to run than the original llm.

Shhh, don’t tell everybody the secret. ;-)
Post reply on HN