Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral AI Launches New 8x22B MOE Model

twitter.com

Post reply on HN
Print view View original

61–70 of 161 posts

Re: Mistral AI Launches New 8x22B MOE Model

#61

Post by reissbaker »

hedgehog wrote: post #44
deoxykev wrote: post #38

4 bit quants should require 85GB VRAM, so this will fit nicely on 4x 24G consumer GPUs, plus some leftover for KV cache optimization.

I've found the 2 bit quant of Mixtral 8x7B is usable for some purposes with an 8GB GPU. I'm curious how this new model will work in similar cheap 8-16GB GPU configurations.

16GB will be way too small unfortunately — this has over 3x the param count, so at best you're looking at a 24GB card with extreme 2bit quantization.

Really though if you're just looking to run models personally and not finetune (which requires monstrous amounts of VRAM), Macs are the way to go for this kind of mega model: Macs have unified memory between the GPU and CPU, and you can buy them with a lot of RAM. It'll be cheaper than trying to buy enough GPU VRAM. A Mac Studio with 192GB unified RAM is under $6k — two A6000s will run you over $9k and still only give you 96GB VRAM (and God help you if you try to build the equivalent system out of 4090s or A100s/H100s).

Or just rent the GPU time as needed from cloud providers like RunPod, although that may or may not be what you're looking for.

Re: Mistral AI Launches New 8x22B MOE Model

#63

Post by segmondy »

nazka wrote: post #52

Out of topic but are we now back at the same performance than ChatGPT 4 at the time people said it worked like magic (meaning before the nerf to make it more politically correct but making his performance crash)?

With open models, yes we are at the performance of at least the first release of ChatGPT 4.

Re: Mistral AI Launches New 8x22B MOE Model

#64

Post by qeternity »

deoxykev wrote: post #38

4 bit quants should require 85GB VRAM, so this will fit nicely on 4x 24G consumer GPUs, plus some leftover for KV cache optimization.

4bit should take up less than this, there are quite a few shared parameters between experts.

But unless you’re running bs=1 it will be painful vs 8x GPU as you’re almost certain to be activating most/all of the experts in a batch.

Re: Mistral AI Launches New 8x22B MOE Model

#65

Post by moffkalast »

jart wrote: post #49
fbdab103 wrote: post #27

Earlier quoted context omitted.

I think the llamafile[0] system works the best. Binary works on the command line or launches a mini webserver. Llamafile offers builds of Mixtral-8x7B-Instruct, so presumably they may package this one up as well (potentially a quantized format). You would have to confirm with someone deeper in the ecosystem, but I think you should be able to run this new model as is against a llamafile? [0] https://github.com/Mozilla…

llamafile author here. I'm downloading Mixtral 8x22b right now. I can't say for certain it'll work until I try it, but let's keep our fingers crossed! If not, we'll be shipping a release as soon as possible that gets it working. My recent work optimizing CPU evaluation https://justine.lol/matmul/ may have come at just the right time. Mixtral 8x7b always worked best at Q5_K_M and higher, which is 31GB. So unless you'v…

Correct me if I'm wrong, but in the tests I've run, the matmul optimizations only have an effect if there's no other blas acceleration. If one can at least offload the KV cache to cublas or run with openblas it's not really used, right? At least I didn't see any speedup in with that config when comparing that PR to the main llama.cpp branch.

Re: Mistral AI Launches New 8x22B MOE Model

#66

Post by moffkalast »

zmmmmm wrote: post #10

A pre-Llama3 race for everyone to get their best small models on the table?

262 GB is not exactly small. But yes it seems they're all getting them out the door in case they end up being worse than llama-3 in which case it'll be too embarrassing to release later.

Re: Mistral AI Launches New 8x22B MOE Model

#67

Post by moffkalast »

Jackson__ wrote: post #50
abdullahkhalids wrote: post #12

Why are some of their models open, and others closed? What is their strategy?

My personal speculation is that their closed models are based on other companies' models. For example on EQbench[0], Miqu[1], a leaked continued pretrain based on LLama2, performs extremely similar to the mistral medium model their API offers. Maybe they're thinking it'd be bad PR for them to release models they didn't create from scratch, or there is some contractual obligation preventing the release. [0] https://eq…

That's quite likely, some have also speculated that Mistral 7B got some EU grant funding that stipulated it had to be openly released later, and Mixtral is based on Mistral 7B so it would likely be subject to the same terms. I haven't found any source to substantiate it though.

Re: Mistral AI Launches New 8x22B MOE Model

#68

Post by moralestapia »

Aissen wrote: post #60
moralestapia wrote: post #18

Earlier quoted context omitted.

What could a malicious model do, though? Curse at you?

Exploit a memory safety issue in the tokenizer/or other parts of your LLM infra written in a native language.

??? With weights?

Re: Mistral AI Launches New 8x22B MOE Model

#69

Post by jart »

moffkalast wrote: post #65
jart wrote: post #49

Earlier quoted context omitted.

llamafile author here. I'm downloading Mixtral 8x22b right now. I can't say for certain it'll work until I try it, but let's keep our fingers crossed! If not, we'll be shipping a release as soon as possible that gets it working. My recent work optimizing CPU evaluation https://justine.lol/matmul/ may have come at just the right time. Mixtral 8x7b always worked best at Q5_K_M and higher, which is 31GB. So unless you'v…

Correct me if I'm wrong, but in the tests I've run, the matmul optimizations only have an effect if there's no other blas acceleration. If one can at least offload the KV cache to cublas or run with openblas it's not really used, right? At least I didn't see any speedup in with that config when comparing that PR to the main llama.cpp branch.

The code that launches my code (see ggml_compute_forward_mul_mat) comes after CLBLAST, Accelerate, and OpenBLAS. The latter take precedence. So if you're not seeing any speedup in enabling them, it's probably because tinyBLAS has reached terms of equality with the BLAS. It's obviously nowhere near as fast as cuBLAS, but maybe PCIE memory transfer overhead explains it. It also really depends on various other factors, like quantization type. For example, the BLAS doesn't support formats like Q4_0 and tinyBLAS does.

Re: Mistral AI Launches New 8x22B MOE Model

#70

Post by KTibow »

idiliv wrote: post #57
aurareturn wrote: post #55

Might be a dumb question but does this mean this model has 176B params?

In Mixtral 8x7B, the 8 means that the model uses Mixture-of-Experts (MoE) layers with 8 experts. The 7B means that if you were to remove 7 of the 8 experts in each layer, then you would end up with a 7B model (which would have exactly the same architecture as Mistral 7B). Therefore, a 1x7B model has 7B params. An 8x7B model has 1 * 7B + (8-1) * sz_expert params, where sz_expert is some constant value that the MoE lay…

I tried to check this for myself.

I agreed for the first one, (46.3 - 7) / 7 = 5.61b.

The second one doesn't match up, (281 - 22) / 7 = 37b or (140.5 - 22) / 7 = 16.92b. Am I doing something wrong?

Post reply on HN