Viewing profile — cubie

I'm a big fan of their work as well, good shout.

That's awesome to hear! It's been growing a lot in the background, still useful as ever, especially for retrieval/semantic search.

Looks very solid; I'm excited for finetuned variants for retrieval and reranking.

Spot on

Not yet - these are base models, or "foundational models". They're great for molding into different use cases via finetuning, better than common models like BERT, RoBERTa, etc. in …

Beyond what the others have said about 1) ModernBERT-base being 149M parameters vs BERT-base's 110M and 2) most LLMs being decoder-only models, also consider that alternating atten…

On a very high level, for NLP: 1. an encoder takes an input (e.g. text), and turns it into a numerical representation (e.g. an embedding). 2. a decoder takes an input (e.g. text), …

That is exactly correct

By "irrespective of their relevance to the language modeling task", the authors mean that the semantic meaning of the tokens is not important. These 4 tokens can be completely repl…

Various experiments on the recent Window Attention with Attention Sinks/StreamingLLM approach indicate that the approach certainly improves inference fluency of pretrained LLMs, wh…