A Replacement for BERT

#1

Post by cubie » Thu, Dec 19, 2024, 4:53 PM UTC

Re: A Replacement for BERT

#2

Post by jbellis » Thu, Dec 19, 2024, 5:49 PM UTC

Looks great, thanks for training this!

  - Can I fine tune it with SentenceTransformers?
  - I see ColBERT in the benchmarks, is there an answerai-colbert-small-v2 coming soon?

Re: A Replacement for BERT

#3

Post by zelias » Thu, Dec 19, 2024, 5:54 PM UTC

missed opportunity to call it ERNIE

Re: A Replacement for BERT

#4

Post by gunalx » Thu, Dec 19, 2024, 5:54 PM UTC

jbellis wrote: post #2

Looks great, thanks for training this! - Can I fine tune it with SentenceTransformers? - I see ColBERT in the benchmarks, is there an answerai-colbert-small-v2 coming soon?

Seems like it. They even have example training scripts available. https://github.com/AnswerDotAI/ModernBERT/blob/main/examples...

Check out their documentation page linked on the bottom of the article. https://huggingface.co/docs/transformers/main/en/model_doc/m...

Re: A Replacement for BERT

#5

Post by amrrs » Thu, Dec 19, 2024, 5:58 PM UTC

zelias wrote: post #3

missed opportunity to call it ERNIE

I remember back in the day there was an Ernie model

Re: A Replacement for BERT

#6

Post by Arcuru » Thu, Dec 19, 2024, 6:05 PM UTC

I'm not sure I am understanding where exactly this slots in, but isn't this an embedding model? Shouldn't they be comparing it to a service like Voyage AI?

- https://docs.voyageai.com/docs/embeddings

Re: A Replacement for BERT

#7

Post by pantsforbirds » Thu, Dec 19, 2024, 6:16 PM UTC

Awesome news and something I really want to checkout for work. Has anyone seen any RAG evals for ModernBERT yet?

Re: A Replacement for BERT

#8

Post by lrog » Thu, Dec 19, 2024, 6:27 PM UTC

zelias wrote: post #3

missed opportunity to call it ERNIE

yep, too late: https://huggingface.co/docs/transformers/en/model_doc/ernie

Re: A Replacement for BERT

#9

Post by axpy906 » Thu, Dec 19, 2024, 6:30 PM UTC

amrrs wrote: post #5

zelias wrote: post #3

missed opportunity to call it ERNIE

I remember back in the day there was an Ernie model

Don’t forget ELMO. The bi-lstm.

Re: A Replacement for BERT

#10

Post by jph00 » Thu, Dec 19, 2024, 6:40 PM UTC

Hi gang, Jeremy from Answer.AI here. Nice to see this on HN! :) We're very excited about this model release -- it feels like it could be the basis of all kinds of interesting new startups and projects.

In fact, the stuff mentioned in the blog post is only the tip of the iceberg. There's a lot of opportunities to fine tune the model in all kinds ways, which I expect will go far beyond what we've managed to achieve in our limited exploration so far.

Anyhoo, if anyone has any questions, feel free to ask!