Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral "Mixtral" 8x7B 32k model [magnet]

twitter.com

Post reply on HN
Print view View original

151–160 of 255 posts

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#151

Post by emporas »

MyFirstSass wrote: post #59

Hot take but Mistral 7B is the actual state of the art of LLM's. ChatGPT 4 is amazing yes and i've been a day 1 subscriber, but it's huge, runs on server farms far away and is more or less a black box. Mistral is tiny, and amazingly coherent and useful for it's size for both general questions and code, uncensored, and a leap i wouldn't have believed possible in just a year. I can run it on my Macbook Air at 12tkps, c…

Given that 50% of all information consumed in the internet is produced in the last 24 hours, smaller models could hold a serious advantage over bigger models.

If an LLM or a SmallLM can be retrained or fine-tuned constantly, every week or every day to incorporate recent information then outdated models trained a year or two years back hold no chance to keep up. Dunno about the licensing but OpenAI could incorporate a smaller model like Mistral7B into their GPT stack, re-train it from scratch every week, and charge the same as GPT-4. There are users who might certainly prefer the weaker, albeit updated models.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#152

Post by brucethemoose2 »

fblgit wrote: post #145
brucethemoose2 wrote: post #57

In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good: https://huggingface.co/fblgit/una-xaberius-34b-v1beta https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16 I mention this because it could theoretically be applied to Mistral Moe. If…

Correct. UNA can align the MoE at multiple layers, experts, nearly any part of the neural network I would say. Xaberius 34B v1 "BETA".. is the king, and its just that.. the beta. I'll be focusing on the Mixtral, its a christmas gift.. modular in that way, thanks for the lab @mistral!

Do a Yi 200K version as well! That would make my Christmas, as Mistral Moe is only maybe 32K.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#153

Post by swyx »

brucethemoose2 wrote: post #57

In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good: https://huggingface.co/fblgit/una-xaberius-34b-v1beta https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16 I mention this because it could theoretically be applied to Mistral Moe. If…

what is neural alignment? who came up with it?

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#154

Post by eurekin »

polygamous_bat wrote: post #147
maremmano wrote: post #111

Earlier quoted context omitted.

Frankly I don't know why Google continues to act this way. Let's remind the "Google Duplex: A.I. Assistant Calls Local Businesses To Make Appointments" story. https://www.youtube.com/watch?v=D5VN56jQMWM Not that this affects Google's user base in any way, at the moment.

> Frankly I don't know why Google continues to act this way. Unfortunately, that's because they have Wall St. analysts looking at their videos who will (indirectly) determine how big of a bonus Sundar and co takes home at the end of the year. Mistral doesn't have to worry about that.

This makes so much sense! Thanks

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#155

Post by nabakin »

MyFirstSass wrote: post #59

Hot take but Mistral 7B is the actual state of the art of LLM's. ChatGPT 4 is amazing yes and i've been a day 1 subscriber, but it's huge, runs on server farms far away and is more or less a black box. Mistral is tiny, and amazingly coherent and useful for it's size for both general questions and code, uncensored, and a leap i wouldn't have believed possible in just a year. I can run it on my Macbook Air at 12tkps, c…

Not a hot take, I think you're right. If it was scaled up to 70b, I think it would be better than Llama 2 70b. Maybe if it was then scaled up to 180b and turned into a MoE it would be better than GPT-4.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#156

Post by filterfiber »

miven wrote: post #76
kcorbitt wrote: post #6

No public statement from Mistral yet. What we know: - Mixture of Experts architecture. - 8x 7B parameters experts (potentially trained starting with their base 7B model?). - 96GB of weights. You won't be able to run this on your home GPU.

> You won't be able to run this on your home GPU. As far as I understand in a MOE model only one/few experts are actually used at the same time, shouldn't the inference speed for this new MOE model be roughly the same as for a normal Mistral 7B then? 7B models have a reasonable throughput when ran on a beefy CPU, especially when quantized down to 4bit precision, so couldn't Mixtral be comfortably ran on a CPU too the…

So this specific model ships with a default config of 2 experts per token.

So you need roughly two loaded in memory per token. Roughly the speed and memory of a 13B per token.

Only issues is that's per-token. 2 experts are choosen per token, which means if they aren't the same ones as the last token, you need to load them into memory.

So yeah to not be disk limited you'd need roughly 8 times the memory and it would run at the speed of a 13B model.

~~~Note on quantization, iirc smaller models lose more performance when quantized vs larger models. So this would be the speed of a 4bit 13B model but with the penalty from a 4bit 7B model.~~~ Actually I have zero idea how quantization scales for MoE, I imagine it has the penalty I mentioned but that's pure speculation.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#157

Post by mrandish »

oh_sigh wrote: post #143
peanuty1 wrote: post #118

Earlier quoted context omitted.

Google Bard is still not available in Canada.

Are there some regulatory reasons why it would not be available? It seems weird if Google would intentionally block users merely to block them.

I think there are still some pretty onerous laws about French localization of products and services made available in the French-speaking part of Canada. Could be that...

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#158

Post by brucethemoose2 »

baq wrote: post #144
brucethemoose2 wrote: post #114

Earlier quoted context omitted.

Which is outrageously priced, in case thats not clear. Its an 2020 RTX 3090 with doubled up memory ICs, which is not much extra BoM.

Clearly it’s worth what people are willing to pay for it. At least it isn’t being used to compute hashes of virtual gold.

Its a artificial supply constraint due to artificial market segmentation enabled by Nvidia/AMD.

Honestly its crazy that AMD indulges in this, especially now. Their workstation market share is comparatively tiny, and instead they could have a swarm of devs (like me) pecking away at AMD compatibility on AI repos if they sold cheap 32GB/48GB cards.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#160

Post by valine »

fblgit wrote: post #142
brucethemoose2 wrote: post #103

Earlier quoted context omitted.

I'm not saying its better than 70B, just that its very strong from what others are saying. Actually I am testing the 34B myself (not the 7B), and it seems good.

UNA: Uniform Neural Alignment. Haven't u noticed yet? Each model that I uniform, behaves like a pre-trained.. and you likely can fine-tune it again without damaging it. If you chatted with them, you know .. that strange sensation, you know what is it.. Intelligence. Xaberius-34B is the highest performer of the board, and is NOT contaminated.

How much data do you need for UNA? Is a typical fine tuning dataset needed or can you get away with less than that?
Post reply on HN