Mistral "Mixtral" 8x7B 32k model [magnet]
twitter.com
Mistral "Mixtral" 8x7B 32k model [magnet]
1–10 of 255 posts
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#2Honest question: Why isn't this on Huggingface? Is this one a leaked model with a questionable training or alignment methodology?
EDIT: I mean, I guess they didn't hack their own twitter account, but still.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#3Still 7B, but now with 32k context. Looking forward to see how it compares with the previous one, and what the community does with it.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#4Looks to be Mixture of Experts, here is the params.json:
{
"dim": 4096,
"n_layers": 32,
"head_dim": 128,
"hidden_dim": 14336,
"n_heads": 32,
"n_kv_heads": 8,
"norm_eps": 1e-05,
"vocab_size": 32000,
"moe": {
"num_experts_per_tok": 2,
"num_experts": 8
}
}Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#586 GB. So it's likely a Mixture of experts model with 8 experts. Exciting.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#6No public statement from Mistral yet. What we know:
- Mixture of Experts architecture.
- 8x 7B parameters experts (potentially trained starting with their base 7B model?).
- 96GB of weights. You won't be able to run this on your home GPU.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#786 GB. So it's likely a Mixture of experts model with 8 experts. Exciting.
Damn, I was hoping it was still a single 7B model that I would be able to run on my GPU
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#8Honest question: Why isn't this on Huggingface? Is this one a leaked model with a questionable training or alignment methodology? EDIT: I mean, I guess they didn't hack their own twitter account, but still.
It'll be on Huggingface soon. This is how they dropped their original 7B model as well. It's a marketing thing, but it works!
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#9Honest question: Why isn't this on Huggingface? Is this one a leaked model with a questionable training or alignment methodology? EDIT: I mean, I guess they didn't hack their own twitter account, but still.
It'll be on Huggingface soon. This is how they dropped their original 7B model as well. It's a marketing thing, but it works!
Ah, well, ok. I appreciate the torrent link -- much faster distribution.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#10No public statement from Mistral yet. What we know: - Mixture of Experts architecture. - 8x 7B parameters experts (potentially trained starting with their base 7B model?). - 96GB of weights. You won't be able to run this on your home GPU.
Theoretically it could fit into a single 24GB GPU if 4-bit quantized. Exllama v2 has even more efficient quantization algorithm, and was able to fit 70B models in 24GB gpu, but only with 2048 tokens of context.