Mistral-8x7B-Chat

#1

Post by birriel » Sun, Dec 10, 2023, 8:13 PM UTC

Re: Mistral-8x7B-Chat

#2

Post by antirez » Sun, Dec 10, 2023, 9:00 PM UTC

llama.cpp support needed ASAP, not just quantized, but I hope it will be albe to run in a way that keeps the same two experts for minimum N tokens generated and re-valuate the expert to select only then, so that it will be possible to do the inference in low ram doing something similar to "paging".

Re: Mistral-8x7B-Chat

#3

Post by a_wild_dandan » Sun, Dec 10, 2023, 9:05 PM UTC

PSA: This is broken for Apple silicon. It relies on the CUDA-only `flash_attn` package. Alas, I wanted to try Mistral's MoE on my MacBook. :( Understandable though! It's a new model with an incredibly bizarre release. My phalanges are crossed.

Re: Mistral-8x7B-Chat

#4

Post by kristianp » Sun, Dec 10, 2023, 9:10 PM UTC

antirez wrote: post #2

llama.cpp support needed ASAP, not just quantized, but I hope it will be albe to run in a way that keeps the same two experts for minimum N tokens generated and re-valuate the expert to select only then, so that it will be possible to do the inference in low ram doing something similar to "paging".

There is an issue for it:

https://github.com/ggerganov/llama.cpp/issues/4381

Re: Mistral-8x7B-Chat

#5

Post by lannisterstark » Sun, Dec 10, 2023, 9:23 PM UTC

Man this LLM stuff gets released faster than I can keep up.

Is there a centralized list somewhere that tests "use this for x purpose, use that for y?"

Re: Mistral-8x7B-Chat

#6

Post by simonw » Sun, Dec 10, 2023, 9:29 PM UTC

I thought the new MoE Mistral model was called Mixtral.

Re: Mistral-8x7B-Chat

#7

Post by brucethemoose2 » Sun, Dec 10, 2023, 9:57 PM UTC

lannisterstark wrote: post #5

Man this LLM stuff gets released faster than I can keep up. Is there a centralized list somewhere that tests "use this for x purpose, use that for y?"

Even the OSS trainers/finetuners themselves don't really know the use case for their models. They just hot drop them and move on, lol.

Re: Mistral-8x7B-Chat

#8

Post by lab » Sun, Dec 10, 2023, 10:01 PM UTC

We just made it available on OpenRouter:

https://openrouter.ai/models/fireworks/mixtral-8x7b-fw-chat

Chat playground:

https://openrouter.ai/playground?models=fireworks/mixtral-8x...

Re: Mistral-8x7B-Chat

#9

Post by rgbrgb » Sun, Dec 10, 2023, 10:02 PM UTC

kristianp wrote: post #4

antirez wrote: post #2

llama.cpp support needed ASAP, not just quantized, but I hope it will be albe to run in a way that keeps the same two experts for minimum N tokens generated and re-valuate the expert to select only then, so that it will be possible to do the inference in low ram doing something similar to "paging".

There is an issue for it: https://github.com/ggerganov/llama.cpp/issues/4381

what a sick project to be able to attract a billionaire programmer [0] and c royalty.

[0]: https://github.com/ggerganov/llama.cpp/issues/4216#issuecomm...

Re: Mistral-8x7B-Chat

#10

Post by Jimbabwe » Sun, Dec 10, 2023, 10:06 PM UTC

There’s probably a better place to ask this highly specific technical question, but I’m avoiding Reddit these days so just throwing it out I guess. I’ve been trying to run these in a container but it’s verrrry slow, I believe, because of the lack of gpu help. All the instructions I find are for nvidia gpus and my server is a qnap tvs-473e with an embedded amd cpu/gpu (I know, I know). The only good news is that I’ve upgraded the ram to 32gb, and I have a 1TB ssd. Any idea of how I can get my own self-hosted LLM/chat service on this funky hardware? The nvidia/docker option requires installing the nvidia runtime alongside docker, but I can’t find an amd equivalent. Thanks. Sorry for the wall of text nobody cares about.