Right on time as LLama 3 is released.
Mistral AI Launches New 8x22B MOE Model
21–30 of 161 posts
Re: Mistral AI Launches New 8x22B MOE Model
#22Earlier quoted context omitted.
At 3:30a France local? Alrighty. I still wait a lil bit ;)
What could a malicious model do, though? Curse at you?
Re: Mistral AI Launches New 8x22B MOE Model
#23Re: Mistral AI Launches New 8x22B MOE Model
#24What's the easiest way to run this assuming that you have the weights and the hardware? Even if it's offloading half of the model to RAM, what tool do you use to load this? Ollama? Llama.cpp? Or just import it with some Python library?
Also, what's the best way to benchmark a model to compare it with others? Are there any tools to use off-the-shelf to do that?
Re: Mistral AI Launches New 8x22B MOE Model
#25Re: Mistral AI Launches New 8x22B MOE Model
#26What's the easiest way to run this assuming that you have the weights and the hardware? Even if it's offloading half of the model to RAM, what tool do you use to load this? Ollama? Llama.cpp? Or just import it with some Python library? Also, what's the best way to benchmark a model to compare it with others? Are there any tools to use off-the-shelf to do that?
The easiest is to use vllm (https://github.com/vllm-project/vllm) to run it on a Couple of A100's, and you can benchmark this using this library (https://github.com/EleutherAI/lm-evaluation-harness)
Re: Mistral AI Launches New 8x22B MOE Model
#27What's the easiest way to run this assuming that you have the weights and the hardware? Even if it's offloading half of the model to RAM, what tool do you use to load this? Ollama? Llama.cpp? Or just import it with some Python library? Also, what's the best way to benchmark a model to compare it with others? Are there any tools to use off-the-shelf to do that?
I think the llamafile[0] system works the best. Binary works on the command line or launches a mini webserver. Llamafile offers builds of Mixtral-8x7B-Instruct, so presumably they may package this one up as well (potentially a quantized format).
You would have to confirm with someone deeper in the ecosystem, but I think you should be able to run this new model as is against a llamafile?
Re: Mistral AI Launches New 8x22B MOE Model
#28To this day 8x7b Mixtral remains the best model you can run on a single 48GB GPU. This has the potential to become the best model you can run on two such GPUs, or on an MBP with maxed out RAM, when 4-bit quantized.
Re: Mistral AI Launches New 8x22B MOE Model
#29To this day 8x7b Mixtral remains the best model you can run on a single 48GB GPU. This has the potential to become the best model you can run on two such GPUs, or on an MBP with maxed out RAM, when 4-bit quantized.
My first thought was how much RAM? Will it work on 64GB M1?
Re: Mistral AI Launches New 8x22B MOE Model
#30What's the easiest way to run this assuming that you have the weights and the hardware? Even if it's offloading half of the model to RAM, what tool do you use to load this? Ollama? Llama.cpp? Or just import it with some Python library? Also, what's the best way to benchmark a model to compare it with others? Are there any tools to use off-the-shelf to do that?
I think the llamafile[0] system works the best. Binary works on the command line or launches a mini webserver. Llamafile offers builds of Mixtral-8x7B-Instruct, so presumably they may package this one up as well (potentially a quantized format). You would have to confirm with someone deeper in the ecosystem, but I think you should be able to run this new model as is against a llamafile? [0] https://github.com/Mozilla…
+1 on llamafile. You can point it to a custom model.