Mistral NeMo
mistral.ai
Mistral NeMo
1–10 of 165 posts
Re: Mistral NeMo
#2Does it have any relation to Nvidia's Nemo? Otherwise, it's unfortunate naming
Re: Mistral NeMo
#3> Today, we are excited to release Mistral NeMo, a 12B model built in collaboration with NVIDIA. Mistral NeMo offers a large context window of up to 128k tokens. Its reasoning, world knowledge, and coding accuracy are state-of-the-art in its size category. As it relies on standard architecture, Mistral NeMo is easy to use and a drop-in replacement in any system using Mistral 7B.
> We have released pre-trained base and instruction-tuned checkpoints checkpoints under the Apache 2.0 license to promote adoption for researchers and enterprises. Mistral NeMo was trained with quantisation awareness, enabling FP8 inference without any performance loss.
So that's... uniformly an improvement at just about everything, right? Large context, permissive license, should have good perf. The one thing I can't tell is how big 12B is going to be (read: how much VRAM/RAM is this thing going to need). Annoyingly and rather confusingly for a model under Apache 2.0, https://huggingface.co/mistralai/Mistral-Nemo-Instruct-2407 refuses to show me files unless I login and "You need to agree to share your contact information to access this model"... though if it's actually as good as it looks, I give it hours before it's reposted without that restriction, which Apache 2.0 allows.
Re: Mistral NeMo
#4Does it have any relation to Nvidia's Nemo? Otherwise, it's unfortunate naming
It looks like it was built jointly with nvidia: https://huggingface.co/nvidia/Mistral-NeMo-12B-Instruct
Re: Mistral NeMo
#5Is "Parameter Creep" going to becomes a thing? They hold up Llama-8b as a competitor despite NeMo having 50% more parameters.
The same thing happened with gemma-27b, where they compared it to all the 7-9b models.
It seems like an easy way to boost benchmarks while coming off as "small" at first glance.
Re: Mistral NeMo
#6Exciting, I think 12B is the sweet spot for running locally - large enough to be useful, fast enough to run on a decent laptop.
Re: Mistral NeMo
#7Exciting, I think 12B is the sweet spot for running locally - large enough to be useful, fast enough to run on a decent laptop.
How much memory does employing the complete 128k window take, though? I've sadly noticed that it can take a significant amount of VRAM to use a larger context window.
edit: e.g. I wouldn't know the correct parameters for this calculator, but going from 8k window to 128k window goes from 1.5 GB to 23 GB: https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calcul...
Re: Mistral NeMo
#8> Mistral NeMo uses a new tokenizer, Tekken, based on Tiktoken, that was trained on over more than 100 languages, and compresses natural language text and source code more efficiently than the SentencePiece tokenizer used in previous Mistral models.
Does anyone have a good answer why everyone went back to SentencePiece in the first place? Byte-pair encoding (which is what tiktoken uses: https://github.com/openai/tiktoken) was shown to be a more efficient encoding as far back as GPT-2 in 2019.
Re: Mistral NeMo
#9Is "Parameter Creep" going to becomes a thing? They hold up Llama-8b as a competitor despite NeMo having 50% more parameters. The same thing happened with gemma-27b, where they compared it to all the 7-9b models. It seems like an easy way to boost benchmarks while coming off as "small" at first glance.
As written here: https://huggingface.co/nvidia/Mistral-NeMo-12B-Instruct
"It significantly outperforms existing models smaller or similar in size." is a statement that goes in that direction and would allow the comparison of a 1.7T param model with a 7b one
Re: Mistral NeMo
#10Exciting, I think 12B is the sweet spot for running locally - large enough to be useful, fast enough to run on a decent laptop.
IMO Google's Gemma2 27B [1] is the sweet spot for running locally on commodity 16GB VRAM cards.