Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral "Mixtral" 8x7B 32k model [magnet]

twitter.com

Post reply on HN
Print view View original

201–210 of 255 posts

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#201

Post by brucethemoose2 »

swyx wrote: post #153
brucethemoose2 wrote: post #57

In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good: https://huggingface.co/fblgit/una-xaberius-34b-v1beta https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16 I mention this because it could theoretically be applied to Mistral Moe. If…

what is neural alignment? who came up with it?

@fblgit apparently, from earlier in this thread.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#202

Post by eyegor »

3abiton wrote: post #175
brucethemoose2 wrote: post #57

In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good: https://huggingface.co/fblgit/una-xaberius-34b-v1beta https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16 I mention this because it could theoretically be applied to Mistral Moe. If…

HF leaderboards are rarely reflective of real world performance especially in small variations, but nonetheless, this is promising. What are the HW requirements for this latest Mistral7B?

Any 7b can run well (~50 tok/s) on an 8gb gpu if you tune the context size. 13b can sometimes run well but typically you'll end up with a tiny context window or slow inference. For cpu, I wouldn't recommend going above 1.3b unless you don't mind waiting around.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#203

Post by Der_Einzige »

BryanLegend wrote: post #98

Andrej Karpathy's take: New open weights LLM from @MistralAI params.json: - hidden_dim / dim = 14336/4096 => 3.5X MLP expand - n_heads / n_kv_heads = 32/8 => 4X multiquery - "moe" => mixture of experts 8X top 2 Likely related code: https://github.com/mistralai/megablocks-public Oddly absent: an over-rehearsed professional release video talking about a revolution in AI. If people are wondering why there is so much AI…

Also, because EMNLP 2023 is happening right now.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#204

Post by aarushsah »

eyegor wrote: post #202
3abiton wrote: post #175

Earlier quoted context omitted.

HF leaderboards are rarely reflective of real world performance especially in small variations, but nonetheless, this is promising. What are the HW requirements for this latest Mistral7B?

Any 7b can run well (~50 tok/s) on an 8gb gpu if you tune the context size. 13b can sometimes run well but typically you'll end up with a tiny context window or slow inference. For cpu, I wouldn't recommend going above 1.3b unless you don't mind waiting around.

How so? I'm only getting 12 t/s using Mistral in LM Studio.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#205

Post by eyegor »

aarushsah wrote: post #204
eyegor wrote: post #202

Earlier quoted context omitted.

Any 7b can run well (~50 tok/s) on an 8gb gpu if you tune the context size. 13b can sometimes run well but typically you'll end up with a tiny context window or slow inference. For cpu, I wouldn't recommend going above 1.3b unless you don't mind waiting around.

How so? I'm only getting 12 t/s using Mistral in LM Studio.

The lazy way is to use text-generation-webui, use an exllamav2 conversion of your model, and turn down context length until it fits (and tick the 8 bit cache option). If you go over your vram it will cut your speed substantially. Like 60/s down to 15/s for an extra 500 context length over what fits. Similar idea applies to any other backends, but you need to shove all the layers into vram if you want decent tok/s. To give you a starting point, typically for 7b models I have to use 4k-6k context length and I use 4-6 bit quantizations for an 8gb gpu. So start at 4 bit, 4k context and adjust up as you can.

You can find most popular models converted for you on huggingface.co if you add exl2 to your search and start with the 4 bit quantized version. Don't bother going above 6 bits even if you have spare vram, practically it doesn't offer much benefit.

For reference I max out around 60 tok/s at 4bit, 50 tok/s at 5bit, 40 at 6bit for some random 7b parameter model on a rtx 2070.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#206

Post by mark_l_watson »

eurekin wrote: post #176
brucethemoose2 wrote: post #57

In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good: https://huggingface.co/fblgit/una-xaberius-34b-v1beta https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16 I mention this because it could theoretically be applied to Mistral Moe. If…

I just played with 7b version. It really feels different than anything I tried before. It could explain a docker compose file. It generated a simple vue application component. I asked around a bit about the example and it was strangely coherent and focused across the whole conversation. It was really well detecting, where I'm starting a new thread (without clearing a context) or referring to things before. It caught…

How did you run it? Are there model files in Ollama format? Are you running on NVidia or Apple Silicon?

EDIT: just saw this “ Megatron (1, 2, and 3) is a large, powerful transformer developed by the Applied Deep Learning Research team at NVIDIA.”

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#207

Post by dkarras »

MacsHeadroom wrote: post #116
refulgentis wrote: post #83

Earlier quoted context omitted.

This is extremely misleading. source: been working in local LLMs since 10 months ago. Got my Mac laptop too. I'm bullish too. But we shouldn't breezily dismiss those concerns out of hand. In practice, it's single digit tokens a second on a $4500 laptop for a model with weights half this size (Llama 2 70B Q2 GGUF => 29 GB, Q8 => 36 GB)

Mixtral 8x7b only needs 12B of weights in RAM per generation. 2B for the attention head and 5B from each of 2 experts. It should be able to run slightly faster than a 13B desnse model, in as little as 16GB of RAM with room to spare.

no doesn't work that way. experts can change per token so for interactive speeds you need all in memory unless you want to wait for model swaps between tokens.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#208

Post by userbinator »

asolidtime1 wrote: post #124

https://huggingface.co/someone13574/mixtral-8x7b-32kseqlen/b... Holy shit, this is some clever marketing. Kinda wonder if any of their employees were part of the warez scene at some point.

They certainly got that aesthetic right; the only thing that stands out (but might be a necessity) is using real names instead of handles.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#209

Post by brucethemoose2 »

eurekin wrote: post #176
brucethemoose2 wrote: post #57

In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good: https://huggingface.co/fblgit/una-xaberius-34b-v1beta https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16 I mention this because it could theoretically be applied to Mistral Moe. If…

I just played with 7b version. It really feels different than anything I tried before. It could explain a docker compose file. It generated a simple vue application component. I asked around a bit about the example and it was strangely coherent and focused across the whole conversation. It was really well detecting, where I'm starting a new thread (without clearing a context) or referring to things before. It caught…

Yeah, the Yi version is quite something too.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#210

Post by brucethemoose2 »

mark_l_watson wrote: post #206
eurekin wrote: post #176

Earlier quoted context omitted.

I just played with 7b version. It really feels different than anything I tried before. It could explain a docker compose file. It generated a simple vue application component. I asked around a bit about the example and it was strangely coherent and focused across the whole conversation. It was really well detecting, where I'm starting a new thread (without clearing a context) or referring to things before. It caught…

How did you run it? Are there model files in Ollama format? Are you running on NVidia or Apple Silicon? EDIT: just saw this “ Megatron (1, 2, and 3) is a large, powerful transformer developed by the Applied Deep Learning Research team at NVIDIA.”

My recommendation is:

- Exui with exl2 files on good GPUs.

- Koboldcpp with gguf files for small GPUs and Apple silicon.

There are many reasons, but in a nutshell they are the fastest and most VRAM efficient.

I can fit 34Bs with about 75K context on a single 24GB 3090 before the quality drop from quantization really starts to get dramatic.

Post reply on HN