Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral "Mixtral" 8x7B 32k model [magnet]

twitter.com

Post reply on HN
Print view View original

111–120 of 255 posts

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#111

Post by maremmano »

eurekin wrote: post #72
maremmano wrote: post #49

Do you need some fancy announcement? let's do it the 90s way: https://twitter.com/erhartford/status/1733159666417545641/ph...

I find that a way more bold and confident than dropping a obviously manipulated and unrealistic marketing page or video

Frankly I don't know why Google continues to act this way. Let's remind the "Google Duplex: A.I. Assistant Calls Local Businesses To Make Appointments" story. https://www.youtube.com/watch?v=D5VN56jQMWM

Not that this affects Google's user base in any way, at the moment.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#112

Post by brucethemoose2 »

terafo wrote: post #64
shubb wrote: post #41

Earlier quoted context omitted.

>> You won't be able to run this on your home GPU. Would this allow you to run each expert on a cheap commodity GPU card so that instead of using expensive 200GB cards we can use a computer with 8 cheap gaming cards in it?

Yes, but you wouldn't want to do that. You will be able to run that on a single 24gb GPU by the end of this weekend.

Maybe two weekends.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#113

Post by andy_xor_andrew »

MyFirstSass wrote: post #59

Hot take but Mistral 7B is the actual state of the art of LLM's. ChatGPT 4 is amazing yes and i've been a day 1 subscriber, but it's huge, runs on server farms far away and is more or less a black box. Mistral is tiny, and amazingly coherent and useful for it's size for both general questions and code, uncensored, and a leap i wouldn't have believed possible in just a year. I can run it on my Macbook Air at 12tkps, c…

I am with you on this. Mistral 7B is amazingly good. There are finetunes of it (the Intel one, and Berkeley Starling) that feel like they are within throwing distance of gpt3.5T... at only 7B!

I was really hoping for a 13B Mistral. I'm not sure if this MOE will run on my 3090 with 24GB. Fingers crossed that quantization + offloading + future tricks will make it runnable.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#114

Post by brucethemoose2 »

coolspot wrote: post #99
refulgentis wrote: post #83

Earlier quoted context omitted.

This is extremely misleading. source: been working in local LLMs since 10 months ago. Got my Mac laptop too. I'm bullish too. But we shouldn't breezily dismiss those concerns out of hand. In practice, it's single digit tokens a second on a $4500 laptop for a model with weights half this size (Llama 2 70B Q2 GGUF => 29 GB, Q8 => 36 GB)

> $4500 Which is more than a price of RTX A6000 48gb ($4k used on ebay)

Which is outrageously priced, in case thats not clear. Its an 2020 RTX 3090 with doubled up memory ICs, which is not much extra BoM.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#116

Post by MacsHeadroom »

refulgentis wrote: post #83
jlokier wrote: post #63

Earlier quoted context omitted.

> - 96GB of weights. You won't be able to run this on your home GPU. You can these days, even in a portable device running on battery. 96GB fits comfortably in some laptop GPUs released this year.

This is extremely misleading. source: been working in local LLMs since 10 months ago. Got my Mac laptop too. I'm bullish too. But we shouldn't breezily dismiss those concerns out of hand. In practice, it's single digit tokens a second on a $4500 laptop for a model with weights half this size (Llama 2 70B Q2 GGUF => 29 GB, Q8 => 36 GB)

Mixtral 8x7b only needs 12B of weights in RAM per generation.

2B for the attention head and 5B from each of 2 experts.

It should be able to run slightly faster than a 13B desnse model, in as little as 16GB of RAM with room to spare.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#117

Post by MacsHeadroom »

dragonwriter wrote: post #43
MacsHeadroom wrote: post #34

Earlier quoted context omitted.

Not 7B, 8x7B. It will run with the speed of a 7B model while being much smarter but requiring ~24GB of RAM instead of ~4GB (in 4bit).

Given the config parametes posted, its 2 experts per token, so the conputation cost per token should be the cost of the conponent that selects experts + 2× cost of a 7B model.

Ah good catch. Upon even closer examination, the attention layer (~2B params) is shared across experts. So in theory you would need 2B for the attention head + 5B for each of two experts in RAM.

That's a total of 12B, meaning this should be able to be run on the same hardware as 13B models with some loading time between generations.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#118

Post by peanuty1 »

HlessClaudesman wrote: post #48
tananaev wrote: post #31

Earlier quoted context omitted.

I'm sure it's also a marketing move to build a certain reputation. Looks like it's working.

Not geoblocking the entirety of Europe also makes them stand out like a ringmaster amongst clowns.

Google Bard is still not available in Canada.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#119

Post by typon »

stavros wrote: post #88
brucethemoose2 wrote: post #57

In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good: https://huggingface.co/fblgit/una-xaberius-34b-v1beta https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16 I mention this because it could theoretically be applied to Mistral Moe. If…

Aren't LLM benchmarks at best irrelevant, at worst lying, at this point?

Yes. The only thing that is relevant is a hidden benchmark that's never released and run by a trusted third party.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#120

Post by syntaxing »

seydor wrote: post #65

looks like they're too busy being awesome. i need a fake video to understand this! What memory will this need? I guess it won't run on my 12GB of vram "moe": {"num_experts_per_tok": 2, "num_experts": 8} I bet many people will re-discover bittorrent tonight

BitTorrent was the craze when llama was leaked on torrent. Then Facebook started taking down all huggingface repos and a bunch of people transitioned to torrent released temporarily. llama 2 changed all this but it was a fun time.
Post reply on HN