Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral "Mixtral" 8x7B 32k model [magnet]

twitter.com

Post reply on HN
Print view View original

101–110 of 255 posts

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#101

Post by brucethemoose2 »

stavros wrote: post #88
brucethemoose2 wrote: post #57

In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good: https://huggingface.co/fblgit/una-xaberius-34b-v1beta https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16 I mention this because it could theoretically be applied to Mistral Moe. If…

Aren't LLM benchmarks at best irrelevant, at worst lying, at this point?

Yes, absolutely. I was just preaching this.

But its not totally irrelevant. They are still a datapoint to consider with some performance correlation. YMMV, but these models actually seem to be quite good for the size in my initial testing.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#102

Post by MyFirstSass »

tarruda wrote: post #93
MyFirstSass wrote: post #59

Hot take but Mistral 7B is the actual state of the art of LLM's. ChatGPT 4 is amazing yes and i've been a day 1 subscriber, but it's huge, runs on server farms far away and is more or less a black box. Mistral is tiny, and amazingly coherent and useful for it's size for both general questions and code, uncensored, and a leap i wouldn't have believed possible in just a year. I can run it on my Macbook Air at 12tkps, c…

> I can run it on my Macbook Air at 12tkps, can't wait to try this on my desktop. That seems kinda low, are you using Metal GPU acceleration with llama.cpp? I don't have a macbook, but saw some of the llama.cpp benchmarks that suggest it can reach close to 30tk/s with GPU acceleration.

Thanks for the tip. I'm on the M2 Air with 16 GB's of ram.

If anyone has faster than 12tkps on Air's let me know.

I'm using the LM Studio GUI over llama.cpp with the "Apple Metal GPU" option. Increasing CPU threads seemingly does nothing either without metal.

Ram usage hovers at 5.5GB with a q5_k_m of Mistral.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#103

Post by brucethemoose2 »

nabakin wrote: post #97
stavros wrote: post #88

Earlier quoted context omitted.

Aren't LLM benchmarks at best irrelevant, at worst lying, at this point?

More or less. The automated benchmarks themselves can be useful when you weed out the models which are overfitting to them. Although, anyone claiming a 7b LLM is better than a well trained 70b LLM like Llama 2 70b chat for the general case, doesn't know what they are talking about. In the future will it be possible? Absolutely, but today we have no architecture or training methodology which would allow it to be possi…

I'm not saying its better than 70B, just that its very strong from what others are saying.

Actually I am testing the 34B myself (not the 7B), and it seems good.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#105

Post by elcomet »

nabakin wrote: post #97
stavros wrote: post #88

Earlier quoted context omitted.

Aren't LLM benchmarks at best irrelevant, at worst lying, at this point?

More or less. The automated benchmarks themselves can be useful when you weed out the models which are overfitting to them. Although, anyone claiming a 7b LLM is better than a well trained 70b LLM like Llama 2 70b chat for the general case, doesn't know what they are talking about. In the future will it be possible? Absolutely, but today we have no architecture or training methodology which would allow it to be possi…

We can only compare specific training procedures though.

With a 7b and a 70b trained the same way, the 70b should always be better

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#109

Post by cjbprime »

tarruda wrote: post #15
coder543 wrote: post #13

Earlier quoted context omitted.

> 96GB of weights. You won't be able to run this on your home GPU. This seems like a non-sequitur. Doesn't MoE select an expert for each token? Presumably, the same expert would frequently be selected for a number of tokens in a row. At that point, you're only running a 7B model, which will easily fit on a GPU. It will be slower when "swapping" experts if you can't fit them all into VRAM at the same time, but it shou…

I will be super happy if this is true. Even if you can't fit all of them in the VRAM, you could load everything in tmpfs, which at least removes disk I/O penalty.

Just mentioning in case it helps anyone out: Linux already has a disk buffer cache. If you have available RAM, it will hold on to pages that have been read from disk until there is enough memory pressure to remove them (and then it will only remove some of them, not all of them). If you don't have available RAM, then the tmpfs wouldn't work. The tmpfs is helpful if you know better than the paging subsystem about how much you really want this data to always stay in RAM no matter what, but that is also much less flexible, because sometimes you need to burst in RAM usage.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#110

Post by CamperBob2 »

coolspot wrote: post #99
refulgentis wrote: post #83

Earlier quoted context omitted.

This is extremely misleading. source: been working in local LLMs since 10 months ago. Got my Mac laptop too. I'm bullish too. But we shouldn't breezily dismiss those concerns out of hand. In practice, it's single digit tokens a second on a $4500 laptop for a model with weights half this size (Llama 2 70B Q2 GGUF => 29 GB, Q8 => 36 GB)

> $4500 Which is more than a price of RTX A6000 48gb ($4k used on ebay)

How fast does it run on that?
Post reply on HN