Earlier quoted context omitted.
I'm curious about their business model.
Well so far their business model seems to be mostly centered about raising money[1]. I do hope they succeed in becoming a succesful contender against OpenAI. [1] https://www.bloomberg.com/news/articles/2023-12-04/openai-ri...
Mistral "Mixtral" 8x7B 32k model [magnet]
61–70 of 255 posts
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#62Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#63No public statement from Mistral yet. What we know: - Mixture of Experts architecture. - 8x 7B parameters experts (potentially trained starting with their base 7B model?). - 96GB of weights. You won't be able to run this on your home GPU.
> - 96GB of weights. You won't be able to run this on your home GPU.
You can these days, even in a portable device running on battery.
96GB fits comfortably in some laptop GPUs released this year.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#64No public statement from Mistral yet. What we know: - Mixture of Experts architecture. - 8x 7B parameters experts (potentially trained starting with their base 7B model?). - 96GB of weights. You won't be able to run this on your home GPU.
>> You won't be able to run this on your home GPU. Would this allow you to run each expert on a cheap commodity GPU card so that instead of using expensive 200GB cards we can use a computer with 8 cheap gaming cards in it?
Yes, but you wouldn't want to do that. You will be able to run that on a single 24gb GPU by the end of this weekend.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#65looks like they're too busy being awesome. i need a fake video to understand this!
What memory will this need? I guess it won't run on my 12GB of vram
"moe": {"num_experts_per_tok": 2, "num_experts": 8}
I bet many people will re-discover bittorrent tonight
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#66Looks to be Mixture of Experts, here is the params.json: { "dim": 4096, "n_layers": 32, "head_dim": 128, "hidden_dim": 14336, "n_heads": 32, "n_kv_heads": 8, "norm_eps": 1e-05, "vocab_size": 32000, "moe": { "num_experts_per_tok": 2, "num_experts": 8 } }
What does expert mean in this context?
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#67Earlier quoted context omitted.
I'm sure it's also a marketing move to build a certain reputation. Looks like it's working.
Not geoblocking the entirety of Europe also makes them stand out like a ringmaster amongst clowns.
Well they are French after all. They should be geoblocking the USA in response for a bit to make a point lol.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#68No public statement from Mistral yet. What we know: - Mixture of Experts architecture. - 8x 7B parameters experts (potentially trained starting with their base 7B model?). - 96GB of weights. You won't be able to run this on your home GPU.
> - 96GB of weights. You won't be able to run this on your home GPU. You can these days, even in a portable device running on battery. 96GB fits comfortably in some laptop GPUs released this year.
Be a lot cooler if you said what laptop, and how much quantisation you're assuming :)
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#69Looks to be Mixture of Experts, here is the params.json: { "dim": 4096, "n_layers": 32, "head_dim": 128, "hidden_dim": 14336, "n_heads": 32, "n_kv_heads": 8, "norm_eps": 1e-05, "vocab_size": 32000, "moe": { "num_experts_per_tok": 2, "num_experts": 8 } }
What does expert mean in this context?
It means it's 8 7B models in a trench coat in a sense, it runs as fast as a 14B (2 experts at a time apparently) but takes up as much memory as a 40B model (70% * 8 * 7B). There is some process trained into it that chooses which experts to use based on the question posed. GPT 4 is allegedly based on the same architecture, but at 8*222B.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#70Earlier quoted context omitted.
What does expert mean in this context?
It means it's 8 7B models in a trench coat in a sense, it runs as fast as a 14B (2 experts at a time apparently) but takes up as much memory as a 40B model (70% * 8 * 7B). There is some process trained into it that chooses which experts to use based on the question posed. GPT 4 is allegedly based on the same architecture, but at 8*222B.
[deleted]