Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Replacement for BERT

huggingface.co

Post reply on HN
Print view View original

71–77 of 77 posts

Re: A Replacement for BERT

#71

Post by bomewish »

authorfly wrote: post #68
bomewish wrote: post #63

Earlier quoted context omitted.

I can't find any info on whether ModernBERT will handle languages other than English; German, Chinese, Arabic? Any info there would be super helpful.

Probably a multilingual version will be needed, like with BERT and RoBERTa. I should hasten to add for multi language tasks(beyond detection), either simpler methods for tasks like multiple language classification/prediction(e.g. word frequency, BERTopic like approaches or SVMs) or LLMs are generally a better candidate. There are a couple of reasons.. 1) That size (even for the large) is too much for multiple languag…

Looking to do super fast embeddings, basically. A few chinese teams seem to have produced some BERT variants so I’ll look there.

Re: A Replacement for BERT

#72

Post by lethibo »

janalsncm wrote: post #28

> encoder-only models add up to over a billion downloads per month, nearly three times more than decoder-only models This is partially because people using decoders aren’t using huggingface at all (they would use an API call) but also because encoders are the unsung heroes of most serious ML applications. If you want to do any ranking, recommendation, RAG, etc it will probably require an encoder. And typically that m…

[dead]

Re: A Replacement for BERT

#73

Post by bclavie »

geekodour wrote: post #69
jph00 wrote: post #10

Hi gang, Jeremy from Answer.AI here. Nice to see this on HN! :) We're very excited about this model release -- it feels like it could be the basis of all kinds of interesting new startups and projects. In fact, the stuff mentioned in the blog post is only the tip of the iceberg. There's a lot of opportunities to fine tune the model in all kinds ways, which I expect will go far beyond what we've managed to achieve in…

Hi Jeremy, I am trying to navigate the space and trying to understand what fits where. Could you shed some lights on what parts of bge-m3 would modernbert overlap with or would this is comparing apples to oranges? https://huggingface.co/BAAI/bge-m3

Hey! It’s more like comparing apples to apple pie.

BGE-M3 is a fine-tuned embedding models. This means that they’ve taken a base language model, which was trained for just language modeling, then applied further fine-tuning to make it useful for a given application, in this case, retrieval.

ModernBERT is one step back earlier in the pipeline: it’s the language model that application-specific models such as M3 build on.

Re: A Replacement for BERT

#74

Post by martin82 »

llm_trw wrote: post #65
janalsncm wrote: post #28

> encoder-only models add up to over a billion downloads per month, nearly three times more than decoder-only models This is partially because people using decoders aren’t using huggingface at all (they would use an API call) but also because encoders are the unsung heroes of most serious ML applications. If you want to do any ranking, recommendation, RAG, etc it will probably require an encoder. And typically that m…

Encoders are suffering from the curse of all successful AI applications: they work so they are no longer AI. Excited about trying this out, less excited about recalculating a petabyte worth of embedding if it's as good as it looks like it will be. At least I can keep my house warm.

Kinda curious what kind of data you have lying around there and what stack you use to create the embeddings and keep them up to date and how you use then...

Re: A Replacement for BERT

#75

Post by llm_trw »

martin82 wrote: post #74
llm_trw wrote: post #65

Earlier quoted context omitted.

Encoders are suffering from the curse of all successful AI applications: they work so they are no longer AI. Excited about trying this out, less excited about recalculating a petabyte worth of embedding if it's as good as it looks like it will be. At least I can keep my house warm.

Kinda curious what kind of data you have lying around there and what stack you use to create the embeddings and keep them up to date and how you use then...

Officially financial data. Unofficially every textbook and science paper ever published. Email me if you're interested.

Re: A Replacement for BERT

#76

Post by shahjaidev »

The community would benefit a lot from a multilingual ModernBERT. Pretraining on a multilingual corpus is crucial for a ranking/retrieval model to be deployed in many industry settings.Simply extending the vocab and fine tuning the en checkpoint won’t quite work. Any plans to release a multilingual checkpoint ?

Re: A Replacement for BERT

#77

Post by 303bookworm »

Really excited to see this! 2 Questions: 1. Did you try using RTD (Electra like pretraining)? Or did you skip that for reasons of compatability? 2. Why not incorporate jamba like Mamba2 alternating layers?
Post reply on HN