Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral AI Launches New 8x22B MOE Model

twitter.com

Post reply on HN
Print view View original

81–90 of 161 posts

Re: Mistral AI Launches New 8x22B MOE Model

#81

Post by Manabu-eo »

hmottestad wrote: post #79
timschmidt wrote: post #75

Earlier quoted context omitted.

Reasonably priced Epyc systems with up to 12 memory channels and support for several TB of system memory are now available. Used datacenter hardware is even less expensive. They are on par with the memory bandwidth available to any one of the CPU, GPU, or NPU in the highest end Macs, but capable of driving MUCH more memory. And much simpler to run Linux or Windows on.

Do you have any feel for the performance compared to the M3 Max?

LLM inference is mostly memory bound. An 12-channel Epyc Genoa with 4800MT/s DDR5 ram clocks at 460.8 GB/sec. It's more than the 400GB/s of the M3 Max, only part of that accessible for the CPU.

And on the Epyc System you can plug much more memory for when you need larger memory and PCI-E gpus, for when you need less faster memory.

Threadripper PRO is only 8-channel, but with memory overclocking it might reach numbers similar to those too.

Re: Mistral AI Launches New 8x22B MOE Model

#82

Post by LeoPanthera »

hmottestad wrote: post #76
freeqaz wrote: post #24

What's the easiest way to run this assuming that you have the weights and the hardware? Even if it's offloading half of the model to RAM, what tool do you use to load this? Ollama? Llama.cpp? Or just import it with some Python library? Also, what's the best way to benchmark a model to compare it with others? Are there any tools to use off-the-shelf to do that?

LM Studio is a great way to test out LLMs on my MacBook: https://lmstudio.ai/ Really easy to search huggingface for new models to test directly in the app.

Make sure you get the prompt template set correctly, the defaults are wrong for a lot of models.

Re: Mistral AI Launches New 8x22B MOE Model

#83

Post by brandall10 »

hmottestad wrote: post #78
moffkalast wrote: post #66

Earlier quoted context omitted.

262 GB is not exactly small. But yes it seems they're all getting them out the door in case they end up being worse than llama-3 in which case it'll be too embarrassing to release later.

Since it’s a MOE model it will only need to load a few of the 8 sub models into vram in order to answer a query. So it may look large, but I think a quantized model will easily fit on a Mac with 64GB of memory and maybe even a bit fewer bits and it’ll fit into 32GB. I think it might be the end for 24GB 4090 cards though :(

Unless something has changed, it needs to load the full 8 models at the same time. During inference it performs like a 2 x base model.

Mixtral 7B @ 5 bit takes up over 30gb on my M3 Max. That's over 90 for this at the same quantization. Realistically you probably need a 128gb machine to run this with good results.

Re: Mistral AI Launches New 8x22B MOE Model

#84

Post by aydyn »

hedgehog wrote: post #44
deoxykev wrote: post #38

4 bit quants should require 85GB VRAM, so this will fit nicely on 4x 24G consumer GPUs, plus some leftover for KV cache optimization.

I've found the 2 bit quant of Mixtral 8x7B is usable for some purposes with an 8GB GPU. I'm curious how this new model will work in similar cheap 8-16GB GPU configurations.

AFAIK, 2-bit quant leads to too much loss of performance, such that you're better off using a different smaller model altogether. See here:

https://www.reddit.com/r/LocalLLaMA/comments/18ituzh/mixtral...

Re: Mistral AI Launches New 8x22B MOE Model

#85

Post by idiliv »

KTibow wrote: post #70
idiliv wrote: post #57

Earlier quoted context omitted.

In Mixtral 8x7B, the 8 means that the model uses Mixture-of-Experts (MoE) layers with 8 experts. The 7B means that if you were to remove 7 of the 8 experts in each layer, then you would end up with a 7B model (which would have exactly the same architecture as Mistral 7B). Therefore, a 1x7B model has 7B params. An 8x7B model has 1 * 7B + (8-1) * sz_expert params, where sz_expert is some constant value that the MoE lay…

I tried to check this for myself. I agreed for the first one, (46.3 - 7) / 7 = 5.61b. The second one doesn't match up, (281 - 22) / 7 = 37b or (140.5 - 22) / 7 = 16.92b. Am I doing something wrong?

Just tried this again and I also arrive at 16.92B. Not sure what I did wrong the first time, thanks for double-checking this!

Re: Mistral AI Launches New 8x22B MOE Model

#86

Post by hedgehog »

Manabu-eo wrote: post #81
hmottestad wrote: post #79

Earlier quoted context omitted.

Do you have any feel for the performance compared to the M3 Max?

LLM inference is mostly memory bound. An 12-channel Epyc Genoa with 4800MT/s DDR5 ram clocks at 460.8 GB/sec. It's more than the 400GB/s of the M3 Max, only part of that accessible for the CPU. And on the Epyc System you can plug much more memory for when you need larger memory and PCI-E gpus, for when you need less faster memory. Threadripper PRO is only 8-channel, but with memory overclocking it might reach numbers…

I'm curious how the newer consumer Ryzens might fare. With LPDDR5X they have >100 GB/s memory bandwidth and the GPUs have been improved quite a bit (16 TFLOPS FP16 nominal in the 780M). There are likely all kinds of software problems but setting that aside the perf/$ and perf/watt might be decent.

Re: Mistral AI Launches New 8x22B MOE Model

#87

Post by sp332 »

segmondy wrote: post #63
nazka wrote: post #52

Out of topic but are we now back at the same performance than ChatGPT 4 at the time people said it worked like magic (meaning before the nerf to make it more politically correct but making his performance crash)?

With open models, yes we are at the performance of at least the first release of ChatGPT 4.

Could you recommend one or a few in particular?

Re: Mistral AI Launches New 8x22B MOE Model

#88

Post by sp332 »

moralestapia wrote: post #68
Aissen wrote: post #60

Earlier quoted context omitted.

Exploit a memory safety issue in the tokenizer/or other parts of your LLM infra written in a native language.

??? With weights?

Yes, there's a reason weights are now distributed as "safetensors" files. Malicious weights files in the old formats are possible, and while I haven't seen evidence of the new format being exploitable, I wouldn't be surprised if someone figures out how to do it eventually.

Re: Mistral AI Launches New 8x22B MOE Model

#89

Post by sheepscreek »

varunvummadi wrote: post #26
freeqaz wrote: post #24

What's the easiest way to run this assuming that you have the weights and the hardware? Even if it's offloading half of the model to RAM, what tool do you use to load this? Ollama? Llama.cpp? Or just import it with some Python library? Also, what's the best way to benchmark a model to compare it with others? Are there any tools to use off-the-shelf to do that?

The easiest is to use vllm ( https://github.com/vllm-project/vllm ) to run it on a Couple of A100's, and you can benchmark this using this library ( https://github.com/EleutherAI/lm-evaluation-harness )

In that regard, it’s even easier to use one Apple Studio with sufficient RAM and llama.cpp or even PyTorch for inference.

Re: Mistral AI Launches New 8x22B MOE Model

#90

Post by cyanydeez »

skissane wrote: post #39
varunvummadi wrote: post #9

Earlier quoted context omitted.

It beats the old GPT4 version in lmsys benchmark you can check it out here https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... but Command R is commercially licensed We can assume that mistral will do a better job.

> but Command R is commercially licensed It is licensed under CC-BY-NC-4.0. That license means you are free to use, modify and redistribute it, so long as you aren't doing so "commercially". What exactly counts as "commercial" use is a complex legal question, and the answer may vary from jurisdiction to jurisdiction (different courts may interpret the phrase differently). But, for example, if you are just using it at…

The His website tends to move towards things that can make money.

That's typically synonymous with commercial.

Post reply on HN