111–120 of 161 posts
Earlier quoted context omitted.
My first thought was how much RAM? Will it work on 64GB M1?
Nope. Just the weights would take 88GB at 4 bit. 128GB MBP ought to be able to run it. If I were to guess, a version for Apple MLX should be available within a few days, for those of us fortunate enough to own such a thing.
Earlier quoted context omitted.
Since it’s a MOE model it will only need to load a few of the 8 sub models into vram in order to answer a query. So it may look large, but I think a quantized model will easily fit on a Mac with 64GB of memory and maybe even a bit fewer bits and it’ll fit into 32GB. I think it might be the end for 24GB 4090 cards though :(
I think you are an optimist here. I can barely run mixtral-8x-7B on my M2 Pro 32G Mac, but I am grateful to be able to run it at all.
It's weird that more than a day after the weights dropped, there still isn't a proper announcement from Mistral with a model card. Nor is it available on Mistral's own platform.
Earlier quoted context omitted.
It is ~260GB with presumably fp16 weights. Should fit into 64GB at 3-bit quantization (~49GB). Edit: To add to this, I've had good luck getting solid output out of mixtral 8x7b at 3-bit, so that isn't small enough to completely kill the model's quality.
I wonder, can you quantize it yourself with some tool?
https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#qu...
Earlier quoted context omitted.
> but Command R is commercially licensed It is licensed under CC-BY-NC-4.0. That license means you are free to use, modify and redistribute it, so long as you aren't doing so "commercially". What exactly counts as "commercial" use is a complex legal question, and the answer may vary from jurisdiction to jurisdiction (different courts may interpret the phrase differently). But, for example, if you are just using it at…
I am not a lawyer, but lately, I have been wondering whether the contra proferentem rule interacts with these licenses.
What's the easiest way to run this assuming that you have the weights and the hardware? Even if it's offloading half of the model to RAM, what tool do you use to load this? Ollama? Llama.cpp? Or just import it with some Python library? Also, what's the best way to benchmark a model to compare it with others? Are there any tools to use off-the-shelf to do that?
I’m sure they are already working on it.
Here is the unofficial benchmark: https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1/...