Is this Mistral large?
Unlikely, this model has a max sequence length of 65k, while mistral large is 32k.
Mistral AI Launches New 8x22B MOE Model
51–60 of 161 posts
Re: Mistral AI Launches New 8x22B MOE Model
#52Out of topic but are we now back at the same performance than ChatGPT 4 at the time people said it worked like magic (meaning before the nerf to make it more politically correct but making his performance crash)?
Re: Mistral AI Launches New 8x22B MOE Model
#534 bit quants should require 85GB VRAM, so this will fit nicely on 4x 24G consumer GPUs, plus some leftover for KV cache optimization.
I've found the 2 bit quant of Mixtral 8x7B is usable for some purposes with an 8GB GPU. I'm curious how this new model will work in similar cheap 8-16GB GPU configurations.
Wouldn't expect that to work at all.
Re: Mistral AI Launches New 8x22B MOE Model
#54To this day 8x7b Mixtral remains the best model you can run on a single 48GB GPU. This has the potential to become the best model you can run on two such GPUs, or on an MBP with maxed out RAM, when 4-bit quantized.
I hope i get it to run on my 96gb m2 in q4.
Re: Mistral AI Launches New 8x22B MOE Model
#55Might be a dumb question but does this mean this model has 176B params?
Re: Mistral AI Launches New 8x22B MOE Model
#56Might be a dumb question but does this mean this model has 176B params?
8x7 had 46B or so.
Re: Mistral AI Launches New 8x22B MOE Model
#57Might be a dumb question but does this mean this model has 176B params?
In Mixtral 8x7B, the 8 means that the model uses Mixture-of-Experts (MoE) layers with 8 experts. The 7B means that if you were to remove 7 of the 8 experts in each layer, then you would end up with a 7B model (which would have exactly the same architecture as Mistral 7B). Therefore, a 1x7B model has 7B params. An 8x7B model has 1 * 7B + (8-1) * sz_expert params, where sz_expert is some constant value that the MoE layers increase by when adding one expert. In the case of Mixtral 8x7B the model size is 46.3GB, so, sz_expert ≈ 5.6B.
If these assumptions port over to 8x22B, then 8x22B has, at 281GB, sz_expert ≈ 13.8B.
Re: Mistral AI Launches New 8x22B MOE Model
#58Might be a dumb question but does this mean this model has 176B params?
In Mixtral 8x7B, the 8 means that the model uses Mixture-of-Experts (MoE) layers with 8 experts. The 7B means that if you were to remove 7 of the 8 experts in each layer, then you would end up with a 7B model (which would have exactly the same architecture as Mistral 7B). Therefore, a 1x7B model has 7B params. An 8x7B model has 1 * 7B + (8-1) * sz_expert params, where sz_expert is some constant value that the MoE lay…
Oh, and to answer your actual question: Assuming that the model is released with 16 bits per parameter, then it as 281GB / 16 bit = 140.5 parameters.
Re: Mistral AI Launches New 8x22B MOE Model
#59Might be a dumb question but does this mean this model has 176B params?
In Mixtral 8x7B, the 8 means that the model uses Mixture-of-Experts (MoE) layers with 8 experts. The 7B means that if you were to remove 7 of the 8 experts in each layer, then you would end up with a 7B model (which would have exactly the same architecture as Mistral 7B). Therefore, a 1x7B model has 7B params. An 8x7B model has 1 * 7B + (8-1) * sz_expert params, where sz_expert is some constant value that the MoE lay…
[deleted]