Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral "Mixtral" 8x7B 32k model [magnet]

twitter.com

Post reply on HN
Print view View original

221–230 of 255 posts

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#221

Post by ukuina »

MyFirstSass wrote: post #102
tarruda wrote: post #93

Earlier quoted context omitted.

> I can run it on my Macbook Air at 12tkps, can't wait to try this on my desktop. That seems kinda low, are you using Metal GPU acceleration with llama.cpp? I don't have a macbook, but saw some of the llama.cpp benchmarks that suggest it can reach close to 30tk/s with GPU acceleration.

Thanks for the tip. I'm on the M2 Air with 16 GB's of ram. If anyone has faster than 12tkps on Air's let me know. I'm using the LM Studio GUI over llama.cpp with the "Apple Metal GPU" option. Increasing CPU threads seemingly does nothing either without metal. Ram usage hovers at 5.5GB with a q5_k_m of Mistral.

LlamaFile typically outperforms LM Studio and even Ollama.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#222

Post by nabakin »

mistrial9 wrote: post #164
nabakin wrote: post #97

Earlier quoted context omitted.

More or less. The automated benchmarks themselves can be useful when you weed out the models which are overfitting to them. Although, anyone claiming a 7b LLM is better than a well trained 70b LLM like Llama 2 70b chat for the general case, doesn't know what they are talking about. In the future will it be possible? Absolutely, but today we have no architecture or training methodology which would allow it to be possi…

quick to assert authoritative opinion - yet the one word "better" belies the message ? Certainly there is are more dimensions worth including in the rating?

Certainly, there may be aspects of a particular 7b model which could beat another particular 70b model and greater detail into different pros and cons of different models are worth considering but people are trying to rank models and if we're ranking (calling one "better" than another), we might as well do it as accurately as we can since it can be so subjective.

I see too many misleading "NEW 7B MODEL BEATS GPT-4" posts. People test those models a couple of times, come back to the comments section, declare it true, and onlookers know no better than to believe it and in my opinion has led to many people claiming 7b models have gotten as good as Llama 2 70b or GPT-4 when it is not the case when you account for the overfit being exhibited by these models and actually put them to the test via human evaluation.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#224

Post by nabakin »

elcomet wrote: post #105
nabakin wrote: post #97

Earlier quoted context omitted.

More or less. The automated benchmarks themselves can be useful when you weed out the models which are overfitting to them. Although, anyone claiming a 7b LLM is better than a well trained 70b LLM like Llama 2 70b chat for the general case, doesn't know what they are talking about. In the future will it be possible? Absolutely, but today we have no architecture or training methodology which would allow it to be possi…

We can only compare specific training procedures though. With a 7b and a 70b trained the same way, the 70b should always be better

Makes sense

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#225

Post by nabakin »

brucethemoose2 wrote: post #103
nabakin wrote: post #97

Earlier quoted context omitted.

More or less. The automated benchmarks themselves can be useful when you weed out the models which are overfitting to them. Although, anyone claiming a 7b LLM is better than a well trained 70b LLM like Llama 2 70b chat for the general case, doesn't know what they are talking about. In the future will it be possible? Absolutely, but today we have no architecture or training methodology which would allow it to be possi…

I'm not saying its better than 70B, just that its very strong from what others are saying. Actually I am testing the 34B myself (not the 7B), and it seems good.

> I'm not saying its better than 70B, just that its very strong from what others are saying.

Gotcha

> Actually I am testing the 34B myself (not the 7B), and it seems good.

I've heard good things about it

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#226

Post by eurekin »

mark_l_watson wrote: post #206
eurekin wrote: post #176

Earlier quoted context omitted.

I just played with 7b version. It really feels different than anything I tried before. It could explain a docker compose file. It generated a simple vue application component. I asked around a bit about the example and it was strangely coherent and focused across the whole conversation. It was really well detecting, where I'm starting a new thread (without clearing a context) or referring to things before. It caught…

How did you run it? Are there model files in Ollama format? Are you running on NVidia or Apple Silicon? EDIT: just saw this “ Megatron (1, 2, and 3) is a large, powerful transformer developed by the Applied Deep Learning Research team at NVIDIA.”

In the textgeneration web ui on NVidia gpu

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#227

Post by nikvdp »

brucethemoose2 wrote: post #57

In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good: https://huggingface.co/fblgit/una-xaberius-34b-v1beta https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16 I mention this because it could theoretically be applied to Mistral Moe. If…

This piqued my interest so I made an ollama modelfile of it for the smallest variant (from TheBloke's GGUF [1] version). It does indeed seem impressively gpt4-ish for such a small model! Feels more coherent than openhermes2.5-mistral which was my previous goto local llm.

If you have ollama installed you can try it out with `ollama run nollama/una-cybertron-7b-v2`.

[1]: https://huggingface.co/TheBloke/una-cybertron-7B-v2-GGUF

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#228

Post by stolsvik »

behnamoh wrote: post #215
sbierwagen wrote: post #212

Earlier quoted context omitted.

If you don't like machine evaluations, you can take a look at the lmsys chatbot arena. You give a prompt, two chatbots answer anonymously, and you pick which answer is better: https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar... On the human ratings, three different 7B LLMs (Two different Openchat models and a Mistral fine tune) beat a version of GPT-3.5. (The top 9 chatbots are GPT and Claude versions. Te…

I think the current approach — train 7b models and then do MoE on them — is the future. It’ll still be only runnable on high end customer devices. As for 13b + MoE, I don’t think any customer device could handle that in the next couple years.

I have no formal credentials to say this, but intuitively I feel this is obviously wrong. You couldn’t have taken 50 rats brains and “mixed” them and expected the result to produce new science.

For some uninteresting regurgitation, sure. But size - width and depth - seems like an important piece for ability to extract deep understanding of the universe.

Also, MoE, as I understand it, will inherently not be able to glean insight into, nor reason about, and certainly not be able to come up with novel understanding, for cross-expert areas.

I believe size matters, a lot.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#229

Post by moffkalast »

nullc wrote: post #198
rishabhjain1198 wrote: post #173

Earlier quoted context omitted.

In a MoE model with experts_per_token = 2 and each expert having 7B params, after picking the experts it should run as fast as the slowest 7B expert, not a comparable 14B model.

Only assuming it's able to hide the faster one in free parallelism.

My CPU trying its best to run inference: parallelwhat?
Post reply on HN