No public statement from Mistral yet. What we know: - Mixture of Experts architecture. - 8x 7B parameters experts (potentially trained starting with their base 7B model?). - 96GB of weights. You won't be able to run this on your home GPU.
[deleted]
Mistral "Mixtral" 8x7B 32k model [magnet]
11–20 of 255 posts
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#12multimodal? 32k context is pretty impressive, curious to test instructability
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#13No public statement from Mistral yet. What we know: - Mixture of Experts architecture. - 8x 7B parameters experts (potentially trained starting with their base 7B model?). - 96GB of weights. You won't be able to run this on your home GPU.
> 96GB of weights. You won't be able to run this on your home GPU.
This seems like a non-sequitur. Doesn't MoE select an expert for each token? Presumably, the same expert would frequently be selected for a number of tokens in a row. At that point, you're only running a 7B model, which will easily fit on a GPU. It will be slower when "swapping" experts if you can't fit them all into VRAM at the same time, but it shouldn't be catastrophic for performance in the way that being unable to fit all layers of an LLM is. It's also easy to imagine caching the N most recent experts in VRAM, where N is the largest number that still fits into your VRAM.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#14[deleted]
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#15No public statement from Mistral yet. What we know: - Mixture of Experts architecture. - 8x 7B parameters experts (potentially trained starting with their base 7B model?). - 96GB of weights. You won't be able to run this on your home GPU.
> 96GB of weights. You won't be able to run this on your home GPU. This seems like a non-sequitur. Doesn't MoE select an expert for each token? Presumably, the same expert would frequently be selected for a number of tokens in a row. At that point, you're only running a 7B model, which will easily fit on a GPU. It will be slower when "swapping" experts if you can't fit them all into VRAM at the same time, but it shou…
I will be super happy if this is true.
Even if you can't fit all of them in the VRAM, you could load everything in tmpfs, which at least removes disk I/O penalty.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#16No public statement from Mistral yet. What we know: - Mixture of Experts architecture. - 8x 7B parameters experts (potentially trained starting with their base 7B model?). - 96GB of weights. You won't be able to run this on your home GPU.
> 96GB of weights. You won't be able to run this on your home GPU. This seems like a non-sequitur. Doesn't MoE select an expert for each token? Presumably, the same expert would frequently be selected for a number of tokens in a row. At that point, you're only running a 7B model, which will easily fit on a GPU. It will be slower when "swapping" experts if you can't fit them all into VRAM at the same time, but it shou…
however, if you need to swap experts on each token, you might as well run on cpu.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#17Earlier quoted context omitted.
> 96GB of weights. You won't be able to run this on your home GPU. This seems like a non-sequitur. Doesn't MoE select an expert for each token? Presumably, the same expert would frequently be selected for a number of tokens in a row. At that point, you're only running a 7B model, which will easily fit on a GPU. It will be slower when "swapping" experts if you can't fit them all into VRAM at the same time, but it shou…
however, if you need to swap experts on each token, you might as well run on cpu.
> Presumably, the same expert would frequently be selected for a number of tokens in a row
In other words, assuming you ask a coding question and there's a coding expert in the mix, it would answer it completely.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#18Might be the training code related with the model https://github.com/mistralai/megablocks-public/tree/pstock/m...
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#19No public statement from Mistral yet. What we know: - Mixture of Experts architecture. - 8x 7B parameters experts (potentially trained starting with their base 7B model?). - 96GB of weights. You won't be able to run this on your home GPU.
> 96GB of weights. You won't be able to run this on your home GPU. This seems like a non-sequitur. Doesn't MoE select an expert for each token? Presumably, the same expert would frequently be selected for a number of tokens in a row. At that point, you're only running a 7B model, which will easily fit on a GPU. It will be slower when "swapping" experts if you can't fit them all into VRAM at the same time, but it shou…
Someone smarter will probably correct me, but I don’t think that is how MoE works. With MoE, a feed-forward network assesses the tokens and selects the best two of eight experts to generate the next token. The choice of experts can change with each new token. For example, let’s say you have two experts that are really good at answering physics questions. For some of the generation, those two will be selected. But later on, maybe the context suggests you need two models better suited to generate French language. This is a silly simplification of what I understand to be going on.
Re: Mistral "Mixtral" 8x7B 32k model [magnet]
#20Earlier quoted context omitted.
however, if you need to swap experts on each token, you might as well run on cpu.
> Presumably, the same expert would frequently be selected for a number of tokens in a row In other words, assuming you ask a coding question and there's a coding expert in the mix, it would answer it completely.
yes I read that. do you think it's reasonable to assume that the same expert will be selected so consistently that model swapping times won't dominate total runtime?