Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Replacement for BERT

huggingface.co

Post reply on HN
Print view View original

41–50 of 77 posts

Re: A Replacement for BERT

#41

Post by cubie »

pantsforbirds wrote: post #7

Awesome news and something I really want to checkout for work. Has anyone seen any RAG evals for ModernBERT yet?

Not yet - these are base models, or "foundational models". They're great for molding into different use cases via finetuning, better than common models like BERT, RoBERTa, etc. in fact, but like those models, these ModernBERT checkpoints can only do one thing: mask filling.

For other tasks, such as retrieval, we still need people to finetune them for it. The ModernBERT documentation has some scripts for finetuning with Sentence Transformers and PyLate for retrieval: https://huggingface.co/docs/transformers/main/en/model_doc/m... But people still need to make and release these models. I have high hopes for them.

Re: A Replacement for BERT

#43

Post by bertobugreport »

jph00 wrote: post #10

Hi gang, Jeremy from Answer.AI here. Nice to see this on HN! :) We're very excited about this model release -- it feels like it could be the basis of all kinds of interesting new startups and projects. In fact, the stuff mentioned in the blog post is only the tip of the iceberg. There's a lot of opportunities to fine tune the model in all kinds ways, which I expect will go far beyond what we've managed to achieve in…

Trying to fine tune on single rig multi-gpu and it crashes, going back down to 1 GPU fixes and training continues (excited to see its results).

Script is near identical with the one below, updated with new imports;

https://huggingface.co/docs/transformers/en/tasks/token_clas...

Re: A Replacement for BERT

#44

Post by wenc »

Can I ask where BERT models are used in production these days?

I was given to understand that they are a better alternative to LLM type models for specific tasks like topic classification because they are trained to discriminate rather than to generate (plus they are bidirectional so they can “understand” context better through lookahead). But LLMs are pretty strong so I wonder if the difference is negligible?

Re: A Replacement for BERT

#45

Post by janalsncm »

KTibow wrote: post #27
spott wrote: post #11

Earlier quoted context omitted.

Embedding models are frequently based on Bert style models, but Bert models can be finetuned to do a lot more than just embeddings. So an embedding focused finetune of modern Bert should be compared to something like voyageai, but not modern Bert itself.

What are the people who keep downloading Bert doing then? Are they the minority who directly use it for embeddings?

They are probably fine tuning on their own particular downstream tasks, either for embeddings or as a component of a larger model.

Re: A Replacement for BERT

#46

Post by deepsquirrelnet »

I read your paper this morning, and am just thrilled with the work. Love the added local attention layers. I’ve experimented with them for years (lucidrains repo), and was always surprised they didn’t go further. Inference speeds are awesome on this model. Scrapping NSP, awesome. Increased masking, awesome. RoPE and longer context, again, bravo. There’s so many great incremental improvements learned over the years and you guys made so many good decisions here.

I’d love to distill a “ModernTinyBERT”, but it seems a bit more complex with the interleaved layers.

Re: A Replacement for BERT

#47

Post by Dalewyn »

Cthulhu_ wrote: post #18
behnamoh wrote: post #16

Earlier quoted context omitted.

I never liked the names BERT and its derivatives. Of all the names on the world, they chose words that are ugly, specific to one culture, and frankly childish.

Sesame Street has been broadcast in 140 countries; Bert (and Ernie) have been localized to 18 languages, including Arabic, Hindi, Japanese, Hebrew and Chinese, with China having an AI called ERNIE because of course. Or to make an overly worded / researched reply to a petulant comment short, they are very much not specific to one culture.

>they are very much not specific to one culture.

It's American culture.

Or as Civilization would put it: We made them buy our blue jeans and listen to our pop music.

Note: I disagree with all the other points like "ugly" and "childish".

Re: A Replacement for BERT

#48

Post by janalsncm »

EGreg wrote: post #35
janalsncm wrote: post #28

> encoder-only models add up to over a billion downloads per month, nearly three times more than decoder-only models This is partially because people using decoders aren’t using huggingface at all (they would use an API call) but also because encoders are the unsung heroes of most serious ML applications. If you want to do any ranking, recommendation, RAG, etc it will probably require an encoder. And typically that m…

Can you go into detail for those of us who aren't as well versed in the tech? What do the encoders do vs the decoders, in this ecosystem? What are some good links to learn about these concepts on a high level? I find all most of the writing about different layers and architectures a bit arcane and inscrutable, especially when it comes to Attention and Self-Attention with multiple heads.

If you’re interested in learning more, the linked article isn’t a bad place to start.

Re: A Replacement for BERT

#49

Post by Kinrany »

cubie wrote: post #38
EGreg wrote: post #35

Earlier quoted context omitted.

Can you go into detail for those of us who aren't as well versed in the tech? What do the encoders do vs the decoders, in this ecosystem? What are some good links to learn about these concepts on a high level? I find all most of the writing about different layers and architectures a bit arcane and inscrutable, especially when it comes to Attention and Self-Attention with multiple heads.

On a very high level, for NLP: 1. an encoder takes an input (e.g. text), and turns it into a numerical representation (e.g. an embedding). 2. a decoder takes an input (e.g. text), and then extends the text. (There's also encoder-decoders, but I won't go into those) These two simple definitions immediately give information on how they can be used. Decoders are at the heart of text generation models, whereas encoders r…

How much does the choice of the encoder depend on the application?

Re: A Replacement for BERT

#50

Post by SoothingSorbet »

cubie wrote: post #38
EGreg wrote: post #35

Earlier quoted context omitted.

Can you go into detail for those of us who aren't as well versed in the tech? What do the encoders do vs the decoders, in this ecosystem? What are some good links to learn about these concepts on a high level? I find all most of the writing about different layers and architectures a bit arcane and inscrutable, especially when it comes to Attention and Self-Attention with multiple heads.

On a very high level, for NLP: 1. an encoder takes an input (e.g. text), and turns it into a numerical representation (e.g. an embedding). 2. a decoder takes an input (e.g. text), and then extends the text. (There's also encoder-decoders, but I won't go into those) These two simple definitions immediately give information on how they can be used. Decoders are at the heart of text generation models, whereas encoders r…

I still find this explanation confusing because decoder-only transformers still embed the input and you can extract input embeddings from them.

Is there a difference here other than encoder-only transformers being bidirectional and their primary output (rather than a byproduct) are input embeddings? Is there a reason other than that bidirectionality that we use specific encoder-only embedding models instead of just cutting and pasting a decoder-only model's embedding phase?

Post reply on HN