Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Replacement for BERT

huggingface.co

Post reply on HN
Print view View original

31–40 of 77 posts

Re: A Replacement for BERT

#31

Post by spott »

KTibow wrote: post #27
spott wrote: post #11

Earlier quoted context omitted.

Embedding models are frequently based on Bert style models, but Bert models can be finetuned to do a lot more than just embeddings. So an embedding focused finetune of modern Bert should be compared to something like voyageai, but not modern Bert itself.

What are the people who keep downloading Bert doing then? Are they the minority who directly use it for embeddings?

I’m honestly not sure why Bert-based-uncased is so popular… the model isn’t that useful on its own. From their huggingface page:

> You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to be fine-tuned on a downstream task. See the model hub to look for fine-tuned versions of a task that interests you.

> Note that this model is primarily aimed at being fine-tuned on tasks that use the whole sentence (potentially masked) to make decisions, such as sequence classification, token classification or question answering. For tasks such as text generation you should look at model like GPT2.

Re: A Replacement for BERT

#33

Post by LunaSea »

jph00 wrote: post #10

Hi gang, Jeremy from Answer.AI here. Nice to see this on HN! :) We're very excited about this model release -- it feels like it could be the basis of all kinds of interesting new startups and projects. In fact, the stuff mentioned in the blog post is only the tip of the iceberg. There's a lot of opportunities to fine tune the model in all kinds ways, which I expect will go far beyond what we've managed to achieve in…

Hi Jeremy, do you have plans to adapt this model for different languages?

Re: A Replacement for BERT

#35

Post by EGreg »

janalsncm wrote: post #28

> encoder-only models add up to over a billion downloads per month, nearly three times more than decoder-only models This is partially because people using decoders aren’t using huggingface at all (they would use an API call) but also because encoders are the unsung heroes of most serious ML applications. If you want to do any ranking, recommendation, RAG, etc it will probably require an encoder. And typically that m…

Can you go into detail for those of us who aren't as well versed in the tech?

What do the encoders do vs the decoders, in this ecosystem? What are some good links to learn about these concepts on a high level? I find all most of the writing about different layers and architectures a bit arcane and inscrutable, especially when it comes to Attention and Self-Attention with multiple heads.

Re: A Replacement for BERT

#36

Post by newfocogi »

jph00 wrote: post #10

Hi gang, Jeremy from Answer.AI here. Nice to see this on HN! :) We're very excited about this model release -- it feels like it could be the basis of all kinds of interesting new startups and projects. In fact, the stuff mentioned in the blog post is only the tip of the iceberg. There's a lot of opportunities to fine tune the model in all kinds ways, which I expect will go far beyond what we've managed to achieve in…

Thank you so much for doing this work. I expect many NLP projects and organizations are going to benefit from this, and I'm looking forward to all the models that will be derived from this. I'm already imagining the things I might try to build with it over the holiday break.

Tiny feedback maybe you can pass along to whoever maintains the HuggingFace blog — the GTE-en-MLM link is broken.

https://huggingface.co/thenlper/gte-en-mlm-large should be https://huggingface.co/Alibaba-NLP/gte-multilingual-mlm-base

Re: A Replacement for BERT

#37

Post by metanonsense »

spott wrote: post #31
KTibow wrote: post #27

Earlier quoted context omitted.

What are the people who keep downloading Bert doing then? Are they the minority who directly use it for embeddings?

I’m honestly not sure why Bert-based-uncased is so popular… the model isn’t that useful on its own. From their huggingface page: > You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to be fine-tuned on a downstream task. See the model hub to look for fine-tuned versions of a task that interests you. > Note that this model is primarily aimed at being fin…

I am out of the game for a year or so (and was never completely in the game), but back then BERT was the basis for lots of interesting applications. The original Vision Transformer (ViT) was based (or at least inspired by) BERT, it was used for graph transformers, visual language understanding, etc.

Re: A Replacement for BERT

#38

Post by cubie »

EGreg wrote: post #35
janalsncm wrote: post #28

> encoder-only models add up to over a billion downloads per month, nearly three times more than decoder-only models This is partially because people using decoders aren’t using huggingface at all (they would use an API call) but also because encoders are the unsung heroes of most serious ML applications. If you want to do any ranking, recommendation, RAG, etc it will probably require an encoder. And typically that m…

Can you go into detail for those of us who aren't as well versed in the tech? What do the encoders do vs the decoders, in this ecosystem? What are some good links to learn about these concepts on a high level? I find all most of the writing about different layers and architectures a bit arcane and inscrutable, especially when it comes to Attention and Self-Attention with multiple heads.

On a very high level, for NLP:

1. an encoder takes an input (e.g. text), and turns it into a numerical representation (e.g. an embedding).

2. a decoder takes an input (e.g. text), and then extends the text.

(There's also encoder-decoders, but I won't go into those)

These two simple definitions immediately give information on how they can be used. Decoders are at the heart of text generation models, whereas encoders return embeddings with which you can do further computations. For example, if your encoder model is finetuned for it, the embeddings can be fed through another linear layer to give you classes (e.g. token classification like NER, or sequence classification for full texts). Or the embeddings can be compared with cosine similarity to determine the similarity of questions and answers. This is at the core of information retrieval/search (see https://sbert.net/). Such similarity between embeddings can also be used for clustering, etc.

In my humble opinion (but it's perhaps a dated opinion), (encoder-)decoders are for when your output is text (chatbots, summarization, translation), and encoders are for when your output is literally anything else. Embeddings are your toolbox, you can shape them into anything, and encoders are the wonderful providers of these embeddings.

Re: A Replacement for BERT

#39

Post by strangecasts »

spott wrote: post #31
KTibow wrote: post #27

Earlier quoted context omitted.

What are the people who keep downloading Bert doing then? Are they the minority who directly use it for embeddings?

I’m honestly not sure why Bert-based-uncased is so popular… the model isn’t that useful on its own. From their huggingface page: > You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to be fine-tuned on a downstream task. See the model hub to look for fine-tuned versions of a task that interests you. > Note that this model is primarily aimed at being fin…

I think this comes down to the Huggingface libraries defaulting to downloading the model from HF if they cannot locate the weights - so "make your own text classifier" tutorial notebooks default to bert-based-uncased as a "standard" pretrained encoder you can put a classification head on top of and finetune, and in turn people run them in Google Colab and just download another copy of the weights on startup, which counts towards the total

Re: A Replacement for BERT

#40

Post by cubie »

querez wrote: post #22
jph00 wrote: post #10

Hi gang, Jeremy from Answer.AI here. Nice to see this on HN! :) We're very excited about this model release -- it feels like it could be the basis of all kinds of interesting new startups and projects. In fact, the stuff mentioned in the blog post is only the tip of the iceberg. There's a lot of opportunities to fine tune the model in all kinds ways, which I expect will go far beyond what we've managed to achieve in…

Two questions: 1) Going by the Runtime vs GLUE graph, the ModernBERT-Base is roughly as fast as BERT-BAse. Given its architecture (especially Alternating Attention), I'm curious why the model not considerably faster than its predecessor. Any insight you could share on that? 2) Most modern LLMs are Encoder+Decoder model. Why not chop of the decoder of one of these (e.g. a small Llama or Mistral or other liberally-lice…

Beyond what the others have said about 1) ModernBERT-base being 149M parameters vs BERT-base's 110M and 2) most LLMs being decoder-only models, also consider that alternating attention (local vs global) only starts helping once you're processing longer texts. With short texts, local attention is equivalent to global attention. I'm not sure what length was used in the picture, but GLUE is mostly pretty short text.
Post reply on HN