Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Replacement for BERT

huggingface.co

Post reply on HN
Print view View original

51–60 of 77 posts

Re: A Replacement for BERT

#51

Post by mark_l_watson »

I saw this early this morning. About for or five years ago I used BERT models for summarization, etc. BERT seemed like a miracle to me back then.

I am going to wait until Ollama has this in their library, even though consuming HF is straight forward.

The speedup is impressive, but then so are the massive speed improvements for LLMs recently.

Apple has supported BERT models in their SDKs for Apple developers for years, it will be interesting to see how quickly they update to this newer tech.

Re: A Replacement for BERT

#52

Post by int_19h »

timClicks wrote: post #25
zelias wrote: post #3

missed opportunity to call it ERNIE

More generally, using the prefix "Modern" haunts every product name that uses it. Technologies move fast and modern becomes antiquated very quickly.

It'll just get shortened to Mobert in the long run anyway.

Re: A Replacement for BERT

#53

Post by bclavie »

querez wrote: post #22
jph00 wrote: post #10

Hi gang, Jeremy from Answer.AI here. Nice to see this on HN! :) We're very excited about this model release -- it feels like it could be the basis of all kinds of interesting new startups and projects. In fact, the stuff mentioned in the blog post is only the tip of the iceberg. There's a lot of opportunities to fine tune the model in all kinds ways, which I expect will go far beyond what we've managed to achieve in…

Two questions: 1) Going by the Runtime vs GLUE graph, the ModernBERT-Base is roughly as fast as BERT-BAse. Given its architecture (especially Alternating Attention), I'm curious why the model not considerably faster than its predecessor. Any insight you could share on that? 2) Most modern LLMs are Encoder+Decoder model. Why not chop of the decoder of one of these (e.g. a small Llama or Mistral or other liberally-lice…

Hey, Ben here, one of the paper's core author authors. The responses you got were mostly spot on.

For (1), it's because BERT has both noticeably fewer parameters, and we're comparing at short context length (in the interest of providing a broader comparison), so local attention is a lot impactful than it is at the longer context lengths.

For (2), most LLMs are actually decoder-only, so there is no "encoder" here. But also, there's not a lot of LLMs in the ±100M parameter range in the first place!

Re: A Replacement for BERT

#54

Post by bclavie »

newfocogi wrote: post #36
jph00 wrote: post #10

Hi gang, Jeremy from Answer.AI here. Nice to see this on HN! :) We're very excited about this model release -- it feels like it could be the basis of all kinds of interesting new startups and projects. In fact, the stuff mentioned in the blog post is only the tip of the iceberg. There's a lot of opportunities to fine tune the model in all kinds ways, which I expect will go far beyond what we've managed to achieve in…

Thank you so much for doing this work. I expect many NLP projects and organizations are going to benefit from this, and I'm looking forward to all the models that will be derived from this. I'm already imagining the things I might try to build with it over the holiday break. Tiny feedback maybe you can pass along to whoever maintains the HuggingFace blog — the GTE-en-MLM link is broken. https://huggingface.co/thenlpe…

Thank you! We're fixing the link.

Re: A Replacement for BERT

#55

Post by bclavie »

timClicks wrote: post #25
zelias wrote: post #3

missed opportunity to call it ERNIE

More generally, using the prefix "Modern" haunts every product name that uses it. Technologies move fast and modern becomes antiquated very quickly.

We had a bit of a discussion around it, but I figured that 6 years warranted the prefix, and it's easier to remember in the sea of new acronyms popping up everyday.

Besides, PostModernBERT will be there for us for the next generational jump.

Re: A Replacement for BERT

#56

Post by data_ders »

ZQ-Dev8 wrote: post #12
jph00 wrote: post #10

Hi gang, Jeremy from Answer.AI here. Nice to see this on HN! :) We're very excited about this model release -- it feels like it could be the basis of all kinds of interesting new startups and projects. In fact, the stuff mentioned in the blog post is only the tip of the iceberg. There's a lot of opportunities to fine tune the model in all kinds ways, which I expect will go far beyond what we've managed to achieve in…

Jeremy, this is awesome! Personally excited for a new wave of sentence transformers built off ModernBERT. A poster below provided the link to a sample ST training script in the ModernBERT repo, so that's great. Do you expect the ModernBERT STs to carry the same advantages over ModernBERT that BERT STs had over the original BERT? Or would you expect caveats based on ModernBERT's updated architecture and capabilities?

what’s ST stand for here? I googled and only got results for BERT STS (semantic text similarity)

Re: A Replacement for BERT

#58

Post by vietvu »

wenc wrote: post #44

Can I ask where BERT models are used in production these days? I was given to understand that they are a better alternative to LLM type models for specific tasks like topic classification because they are trained to discriminate rather than to generate (plus they are bidirectional so they can “understand” context better through lookahead). But LLMs are pretty strong so I wonder if the difference is negligible?

LLMs like GPT are heavy and costly (and BERT are LLMs too, params can up to like 1.5B). For niche problems like classification on a small domain, BERT like models are much better, cheaper. You don't need all knowledge gen AI LLM has. I have seen many companies using DeBERTa or RoBERTa for text classification, not using GPT/LLaMA.

Re: A Replacement for BERT

#59

Post by bclavie »

data_ders wrote: post #56
ZQ-Dev8 wrote: post #12

Earlier quoted context omitted.

Jeremy, this is awesome! Personally excited for a new wave of sentence transformers built off ModernBERT. A poster below provided the link to a sample ST training script in the ModernBERT repo, so that's great. Do you expect the ModernBERT STs to carry the same advantages over ModernBERT that BERT STs had over the original BERT? Or would you expect caveats based on ModernBERT's updated architecture and capabilities?

what’s ST stand for here? I googled and only got results for BERT STS (semantic text similarity)

Sentence Transformers (https://sbert.net/), the most used library for embedding models (similarity, retrieval.)

Re: A Replacement for BERT

#60

Post by craigacp »

SoothingSorbet wrote: post #50
cubie wrote: post #38

Earlier quoted context omitted.

On a very high level, for NLP: 1. an encoder takes an input (e.g. text), and turns it into a numerical representation (e.g. an embedding). 2. a decoder takes an input (e.g. text), and then extends the text. (There's also encoder-decoders, but I won't go into those) These two simple definitions immediately give information on how they can be used. Decoders are at the heart of text generation models, whereas encoders r…

I still find this explanation confusing because decoder-only transformers still embed the input and you can extract input embeddings from them. Is there a difference here other than encoder-only transformers being bidirectional and their primary output (rather than a byproduct) are input embeddings? Is there a reason other than that bidirectionality that we use specific encoder-only embedding models instead of just c…

The encoder's embedding is contextual, it depends on all the tokens. If you pull out the embedding layer from a decoder only model then that is a fixed embedding where each token's representation doesn't depend on the other tokens in the sequence. The bi-directionality is also important for getting a proper representation of the sequence, though you can train decoder only models to emit a single embedding vector once they have processed the whole sequence left to right.

Fundamentally it's basically a difference between bidirectional attention in the encoder and a triangular (or "causal") attention mask in the decoder.

Post reply on HN