It would be really cool to have a model like this but multilingual, it would really help with things like moderation.
A Replacement for BERT
61–70 of 77 posts
Re: A Replacement for BERT
#62How does it compare to Jina V3 [0], which also has 8192 context length?
Re: A Replacement for BERT
#63Hi gang, Jeremy from Answer.AI here. Nice to see this on HN! :) We're very excited about this model release -- it feels like it could be the basis of all kinds of interesting new startups and projects. In fact, the stuff mentioned in the blog post is only the tip of the iceberg. There's a lot of opportunities to fine tune the model in all kinds ways, which I expect will go far beyond what we've managed to achieve in…
I can't find any info on whether ModernBERT will handle languages other than English; German, Chinese, Arabic? Any info there would be super helpful.
Re: A Replacement for BERT
#64How does it compare to Jina V3 [0], which also has 8192 context length? 0. https://arxiv.org/abs/2409.10173
They perform different roles, so they're not directly comparable.
Jina V3 is an embedding model, so it's a base model, further fine-tuned specifically for embedding-ish tasks (retrieval, similarity...). This is what we call "downstream" models/applications.
ModernBERT is a base model & architecture. It's not supposed to be out of the box, but fine-tuned for other use-cases, serving as their backbone. In theory (and, given early signal, most likely in practice too), it'll make for really good downstream embeddings once people build on top of it!
Re: A Replacement for BERT
#65> encoder-only models add up to over a billion downloads per month, nearly three times more than decoder-only models This is partially because people using decoders aren’t using huggingface at all (they would use an API call) but also because encoders are the unsung heroes of most serious ML applications. If you want to do any ranking, recommendation, RAG, etc it will probably require an encoder. And typically that m…
Encoders are suffering from the curse of all successful AI applications: they work so they are no longer AI.
Excited about trying this out, less excited about recalculating a petabyte worth of embedding if it's as good as it looks like it will be. At least I can keep my house warm.
Re: A Replacement for BERT
#66I read your paper this morning, and am just thrilled with the work. Love the added local attention layers. I’ve experimented with them for years (lucidrains repo), and was always surprised they didn’t go further. Inference speeds are awesome on this model. Scrapping NSP, awesome. Increased masking, awesome. RoPE and longer context, again, bravo. There’s so many great incremental improvements learned over the years an…
> I’d love to distill a “ModernTinyBERT
That’s a question I’m interested in as well! DistilBERT and friends have been terribly useful at the edge. I wonder if/when we may see something similar for ModernBERT.
Re: A Replacement for BERT
#67Sad that it is English only, not multilingual.
Re: A Replacement for BERT
#68Hi gang, Jeremy from Answer.AI here. Nice to see this on HN! :) We're very excited about this model release -- it feels like it could be the basis of all kinds of interesting new startups and projects. In fact, the stuff mentioned in the blog post is only the tip of the iceberg. There's a lot of opportunities to fine tune the model in all kinds ways, which I expect will go far beyond what we've managed to achieve in…
I can't find any info on whether ModernBERT will handle languages other than English; German, Chinese, Arabic? Any info there would be super helpful.
Probably a multilingual version will be needed, like with BERT and RoBERTa. I should hasten to add for multi language tasks(beyond detection), either simpler methods for tasks like multiple language classification/prediction(e.g. word frequency, BERTopic like approaches or SVMs) or LLMs are generally a better candidate.
There are a couple of reasons.. 1) That size (even for the large) is too much for multiple languages with good BLEU scores. 2) Encoder and decoder models don't tend to get trained for translation as much as e.g. GPT models with large translation texts in their datasets across multiple languages (with exceptions such as T5 translation task).
Re: A Replacement for BERT
#69Hi gang, Jeremy from Answer.AI here. Nice to see this on HN! :) We're very excited about this model release -- it feels like it could be the basis of all kinds of interesting new startups and projects. In fact, the stuff mentioned in the blog post is only the tip of the iceberg. There's a lot of opportunities to fine tune the model in all kinds ways, which I expect will go far beyond what we've managed to achieve in…
Hi Jeremy, I am trying to navigate the space and trying to understand what fits where.
Could you shed some lights on what parts of bge-m3 would modernbert overlap with or would this is comparing apples to oranges?
Re: A Replacement for BERT
#70Can I ask where BERT models are used in production these days? I was given to understand that they are a better alternative to LLM type models for specific tasks like topic classification because they are trained to discriminate rather than to generate (plus they are bidirectional so they can “understand” context better through lookahead). But LLMs are pretty strong so I wonder if the difference is negligible?
LLMs dont have the same usecase as encoder only models. Lets assume you have around million keywords and you want to find the most similar to a keyword that the user input.
In pre-processing you would have calculated the vector encoding of all the million keywords before hand and now with the keyword the user input, you calculate the vector and then find the most similar vectors
LLM is used by end user, encoders are used by devs in app to search/retrieve text.