71–80 of 255 posts
Do you need some fancy announcement? let's do it the 90s way: https://twitter.com/erhartford/status/1733159666417545641/ph...
looks like they're too busy being awesome. i need a fake video to understand this! What memory will this need? I guess it won't run on my 12GB of vram "moe": {"num_experts_per_tok": 2, "num_experts": 8} I bet many people will re-discover bittorrent tonight
Its also a good candidate for splitting across small GPUs, maybe.
One architecture I can envision is hosting prompt ingestion and the "host" model on the GPU and the downstream expert model weights on the CPU /IGP. This is actually pretty efficient, as the CPU/IGP is really bad at the prompt ingestion but reasonably fast at ~14B token generation.
Llama.cpp all but already does this, I'm sure MLC will implement it as well.
Still 7B, but now with 32k context. Looking forward to see how it compares with the previous one, and what the community does with it.
No public statement from Mistral yet. What we know: - Mixture of Experts architecture. - 8x 7B parameters experts (potentially trained starting with their base 7B model?). - 96GB of weights. You won't be able to run this on your home GPU.
That is only 24GB in 4bit. People are running models 2-4 times that size on local GPUs. What's more, this will run on a MacBook CPU just fine-- and at an extremely high speed.
This is just about right for 24GB. I bet that is intentional on their part.
No public statement from Mistral yet. What we know: - Mixture of Experts architecture. - 8x 7B parameters experts (potentially trained starting with their base 7B model?). - 96GB of weights. You won't be able to run this on your home GPU.
As far as I understand in a MOE model only one/few experts are actually used at the same time, shouldn't the inference speed for this new MOE model be roughly the same as for a normal Mistral 7B then?
7B models have a reasonable throughput when ran on a beefy CPU, especially when quantized down to 4bit precision, so couldn't Mixtral be comfortably ran on a CPU too then, just with 8 times the memory footprint?
Still 7B, but now with 32k context. Looking forward to see how it compares with the previous one, and what the community does with it.
unfortunately too big for the broader community to test. Will be very interesting to see how well it performs compared to the large models
Earlier quoted context omitted.
I don't see any code in there. What runtime could load these weights?
Its presumably llama just like Mistral. Everything open source is llama now. Facebook all but standardized the architecture. I dunno about the moe. Is there existing transformers code for that part? It kinda looks like there is based on the config.
Earlier quoted context omitted.
I don't see any code in there. What runtime could load these weights?
Its presumably llama just like Mistral. Everything open source is llama now. Facebook all but standardized the architecture. I dunno about the moe. Is there existing transformers code for that part? It kinda looks like there is based on the config.
Earlier quoted context omitted.
> - 96GB of weights. You won't be able to run this on your home GPU. You can these days, even in a portable device running on battery. 96GB fits comfortably in some laptop GPUs released this year.
Be a lot cooler if you said what laptop, and how much quantisation you're assuming :)