Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What happened to BERT and T5?

yitay.net

Post reply on HN
Print view View original

41–50 of 69 posts

Re: What happened to BERT and T5?

#41

Post by hdhshdhshdjd »

Tostino wrote: post #37
hdhshdhshdjd wrote: post #28

Maybe in SOTA ml/nlp research, but in the world of building useful tools and products, BERT models are dead simple to tune, work great if you have decent training data, and most importantly are very very fast and very very cheap to run. I have a small Swiss army collection of custom BERT fine tunes that are equal or better than the best LLM and execute document classification tasks in 2.4ms. Find me an LLM that can d…

Want to share your collection with the class so we can all learn? Seems useful.

Product in stealth for a little bit longer, so can’t say much. :-)

Re: What happened to BERT and T5?

#42

Post by hdhshdhshdjd »

ipsum2 wrote: post #36
hdhshdhshdjd wrote: post #28

Maybe in SOTA ml/nlp research, but in the world of building useful tools and products, BERT models are dead simple to tune, work great if you have decent training data, and most importantly are very very fast and very very cheap to run. I have a small Swiss army collection of custom BERT fine tunes that are equal or better than the best LLM and execute document classification tasks in 2.4ms. Find me an LLM that can d…

What does your swiss army collection do?

Document classification in highly ambiguous contextual space. Solving some specific large scale classification tasks, so multi million document sets.

Re: What happened to BERT and T5?

#43

Post by deepsquirrelnet »

lalaland1125 wrote: post #7

I think the big reason why BERT and T5 have fallen out of favor is the lack of zero shot (or few shot) ability. When you have hundreds or thousands of examples, BERT works great. But that is very restricting.

They are there, you just have to look. Tasksource, NuNER, Flan, T0. There’s not a lot, but still at least a few good zero shot models in both architectures.

Re: What happened to BERT and T5?

#44

Post by deepsquirrelnet »

hdhshdhshdjd wrote: post #28

Maybe in SOTA ml/nlp research, but in the world of building useful tools and products, BERT models are dead simple to tune, work great if you have decent training data, and most importantly are very very fast and very very cheap to run. I have a small Swiss army collection of custom BERT fine tunes that are equal or better than the best LLM and execute document classification tasks in 2.4ms. Find me an LLM that can d…

Latency, throughput and cost are still very important for many applications.

Also the output of a purpose-built encoder model is preferable to natural language. Not only is it unambiguous, but scores are often an important part of the result.

Last, if you need to get into some advanced methods of training, like pseudolabeling and semi-supervised learning, there’s different options and outlets for utilizing real world datasets.

That said, I’m not sure there’s much value in scaling up current encoder models. It seems like there’s already a point of diminishing returns.

Re: What happened to BERT and T5?

#45

Post by Seattle3503 »

hdhshdhshdjd wrote: post #28

Maybe in SOTA ml/nlp research, but in the world of building useful tools and products, BERT models are dead simple to tune, work great if you have decent training data, and most importantly are very very fast and very very cheap to run. I have a small Swiss army collection of custom BERT fine tunes that are equal or better than the best LLM and execute document classification tasks in 2.4ms. Find me an LLM that can d…

What technique do you use to get BERT to work on longer documents?

Re: What happened to BERT and T5?

#46

Post by Grimblewald »

tempusalaria wrote: post #40
lalaland1125 wrote: post #12

Earlier quoted context omitted.

The key to understanding the difference is that transformers are attention models where tokens can "attend" to different tokens. Encoder models allow all tokens to attend to every other token. This increases the number of connections and makes it easier for the model to reason, but requires all tokens at once to produce any output. These models generally can't generate text. Decoder models only allow tokens to attend…

This is wrong. The term for models that look only at previous tokens in the sequence is auto-regressive. Encoder and decoder has nothing to do with this.

arent a lot of transformers built in a way where attention is only applied to previous tokens in sequence, even though its fully possible to apply it both ways?

Re: What happened to BERT and T5?

#48

Post by robrenaud »

jerrygenser wrote: post #23
lalaland1125 wrote: post #7

I think the big reason why BERT and T5 have fallen out of favor is the lack of zero shot (or few shot) ability. When you have hundreds or thousands of examples, BERT works great. But that is very restricting.

Yes but you can use an llm to label data and then train a bert model which then costs a small fraction of time and money to run than the original llm.

Is the encoder style arch better for representing classification tasks at a given compute budget than a causal LM?

Is this because the final represention in bert style models more globally focused, rather than being optimized for next token prediction?

Re: What happened to BERT and T5?

#49

Post by pizza »

Grimblewald wrote: post #46
tempusalaria wrote: post #40

Earlier quoted context omitted.

This is wrong. The term for models that look only at previous tokens in the sequence is auto-regressive. Encoder and decoder has nothing to do with this.

arent a lot of transformers built in a way where attention is only applied to previous tokens in sequence, even though its fully possible to apply it both ways?

That's the autoregressive aspect. The decoder aspect is that the last layer converts representations into output sequences (and the generation happens autoregressively, one at a time). Similarly at the last layer an encoder outputs a representation/embedding (while being able to attend to the entire sequence).

Re: What happened to BERT and T5?

#50

Post by hdhshdhshdjd »

Seattle3503 wrote: post #45
hdhshdhshdjd wrote: post #28

Maybe in SOTA ml/nlp research, but in the world of building useful tools and products, BERT models are dead simple to tune, work great if you have decent training data, and most importantly are very very fast and very very cheap to run. I have a small Swiss army collection of custom BERT fine tunes that are equal or better than the best LLM and execute document classification tasks in 2.4ms. Find me an LLM that can d…

What technique do you use to get BERT to work on longer documents?

512 has been sufficient to solve my problems. I had done some initial attempts with BigBird that weren’t going well, but then realized I didn’t really need it.

I may revisit at some point.

Post reply on HN