Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happened to BERT and T5?

yitay.net

Post reply on HN
Print view View original

31–40 of 69 posts

Re: What happened to BERT and T5?

#31

Post by llm_trw »

hdhshdhshdjd wrote: post #28

Maybe in SOTA ml/nlp research, but in the world of building useful tools and products, BERT models are dead simple to tune, work great if you have decent training data, and most importantly are very very fast and very very cheap to run. I have a small Swiss army collection of custom BERT fine tunes that are equal or better than the best LLM and execute document classification tasks in 2.4ms. Find me an LLM that can d…

Yeah, pretty much. When you have 2b files you need to troll through good luck using anything but a vector database. Once you do a level or two of pruning of the results then you can feed it into an LLM for final classification.

Re: What happened to BERT and T5?

#32

Post by llm_trw »

hdhshdhshdjd wrote: post #29
janalsncm wrote: post #24

BERT didn’t go anywhere and I have seen fine-tuned BERT backbones everywhere. They are useful for generating embeddings to be used downstream, and small enough to be handled on consumer (pre Ampere) hardware. One of the trends I have seen is scaling BERT down rather than up, since BERT already gave good performance, we want to be able to do it faster and cheaper. That gave rise to RoBERTa, ALBERT and distillBERT. T5…

I tried some large scale translation tasks with T5 and results were iffy at best. I’m going to try the same task with the newest Mistral small models and compare. My guess is Mistral will be better.

T5 is not Bert, translation is not embedding.

Re: What happened to BERT and T5?

#33

Post by jszymborski »

> It is also worth to note that, generally speaking, an Encoder-Decoders of 2N parameters has the same compute cost as a decoder-only model of N parameters which gives it a different FLOP to parameter count ratio.

Can someone explain this to me? I'm not sure how the compute costs are the same between the 2N and N nets.

Re: What happened to BERT and T5?

#34

Post by hdhshdhshdjd »

llm_trw wrote: post #32
hdhshdhshdjd wrote: post #29

Earlier quoted context omitted.

I tried some large scale translation tasks with T5 and results were iffy at best. I’m going to try the same task with the newest Mistral small models and compare. My guess is Mistral will be better.

T5 is not Bert, translation is not embedding.

The article mentions T5 and translation is something T5 is supposedly good at - just sharing I was less than impressed.

Re: What happened to BERT and T5?

#35

Post by phillypham »

jszymborski wrote: post #33

> It is also worth to note that, generally speaking, an Encoder-Decoders of 2N parameters has the same compute cost as a decoder-only model of N parameters which gives it a different FLOP to parameter count ratio. Can someone explain this to me? I'm not sure how the compute costs are the same between the 2N and N nets.

You can break your sequence into two parts. One part goes through the encoder and the other goes through the decoder, so each token only goes through one transformer stack.

Re: What happened to BERT and T5?

#36

Post by ipsum2 »

hdhshdhshdjd wrote: post #28

Maybe in SOTA ml/nlp research, but in the world of building useful tools and products, BERT models are dead simple to tune, work great if you have decent training data, and most importantly are very very fast and very very cheap to run. I have a small Swiss army collection of custom BERT fine tunes that are equal or better than the best LLM and execute document classification tasks in 2.4ms. Find me an LLM that can d…

What does your swiss army collection do?

Re: What happened to BERT and T5?

#37

Post by Tostino »

hdhshdhshdjd wrote: post #28

Maybe in SOTA ml/nlp research, but in the world of building useful tools and products, BERT models are dead simple to tune, work great if you have decent training data, and most importantly are very very fast and very very cheap to run. I have a small Swiss army collection of custom BERT fine tunes that are equal or better than the best LLM and execute document classification tasks in 2.4ms. Find me an LLM that can d…

Want to share your collection with the class so we can all learn? Seems useful.

Re: What happened to BERT and T5?

#38

Post by bjourne »

lalaland1125 wrote: post #12
andy_xor_andrew wrote: post #9

I'm a bit embarrassed to admit, but I still don't understand decoder vs encoder vs decoder/encoder models. Is the input/output of these models any different? Are they all just "text context goes in, scores for all tokens in the vocabulary come out" ? Is the difference only in how they achieve this output?

The key to understanding the difference is that transformers are attention models where tokens can "attend" to different tokens. Encoder models allow all tokens to attend to every other token. This increases the number of connections and makes it easier for the model to reason, but requires all tokens at once to produce any output. These models generally can't generate text. Decoder models only allow tokens to attend…

But this has nothing to do with encoding and decoding.

Re: What happened to BERT and T5?

#40

Post by tempusalaria »

lalaland1125 wrote: post #12
andy_xor_andrew wrote: post #9

I'm a bit embarrassed to admit, but I still don't understand decoder vs encoder vs decoder/encoder models. Is the input/output of these models any different? Are they all just "text context goes in, scores for all tokens in the vocabulary come out" ? Is the difference only in how they achieve this output?

The key to understanding the difference is that transformers are attention models where tokens can "attend" to different tokens. Encoder models allow all tokens to attend to every other token. This increases the number of connections and makes it easier for the model to reason, but requires all tokens at once to produce any output. These models generally can't generate text. Decoder models only allow tokens to attend…

This is wrong.

The term for models that look only at previous tokens in the sequence is auto-regressive.

Encoder and decoder has nothing to do with this.

Post reply on HN