Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral "Mixtral" 8x7B 32k model [magnet]

twitter.com

Post reply on HN
Print view View original

231–240 of 255 posts

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#231

Post by moffkalast »

WeMoveOn wrote: post #163
moffkalast wrote: post #69

Earlier quoted context omitted.

It means it's 8 7B models in a trench coat in a sense, it runs as fast as a 14B (2 experts at a time apparently) but takes up as much memory as a 40B model (70% * 8 * 7B). There is some process trained into it that chooses which experts to use based on the question posed. GPT 4 is allegedly based on the same architecture, but at 8*222B.

How did you come up with 40b for the memory? specifically, why 0.7 * total params?

It's just a rough estimate given that these things are fairly linear, the original 7B mistral was 15 GB and the new one is 86 GB, whereas a fully duplicated 8 * 15 GB would suggest a 120 GB size, so 86/120 = 0.71 for actual size, suggesting 29% memory savings. This of course doesn't really account for any multiple vs single file saving overhead and such, so it's likely to be a bit off.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#232

Post by peanuty1 »

ComputerGuru wrote: post #181
oh_sigh wrote: post #143

Earlier quoted context omitted.

Are there some regulatory reasons why it would not be available? It seems weird if Google would intentionally block users merely to block them.

Google and Facebook were, up until just a couple of days ago, in a cold war within the Canadian government.

Google made a deal to pay 100M/year to news organizations in Canada but Meta is continuing to block news links.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#233

Post by throwaway4aday »

tananaev wrote: post #31
mareksotak wrote: post #22

Some companies spend weeks on landing pages, demos and cute thought through promo videos and then there is Mistral, casually dropping a magnet link on Friday.

I'm sure it's also a marketing move to build a certain reputation. Looks like it's working.

technically, it is marketing but at this level marketing is indistinguishable from shipping

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#234

Post by mark_l_watson »

brucethemoose2 wrote: post #210
mark_l_watson wrote: post #206

Earlier quoted context omitted.

How did you run it? Are there model files in Ollama format? Are you running on NVidia or Apple Silicon? EDIT: just saw this “ Megatron (1, 2, and 3) is a large, powerful transformer developed by the Applied Deep Learning Research team at NVIDIA.”

My recommendation is: - Exui with exl2 files on good GPUs. - Koboldcpp with gguf files for small GPUs and Apple silicon. There are many reasons, but in a nutshell they are the fastest and most VRAM efficient. I can fit 34Bs with about 75K context on a single 24GB 3090 before the quality drop from quantization really starts to get dramatic.

Thanks! I will check out Koboldcpp.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#235

Post by baq »

brucethemoose2 wrote: post #158
baq wrote: post #144

Earlier quoted context omitted.

Clearly it’s worth what people are willing to pay for it. At least it isn’t being used to compute hashes of virtual gold.

Its a artificial supply constraint due to artificial market segmentation enabled by Nvidia/AMD. Honestly its crazy that AMD indulges in this, especially now. Their workstation market share is comparatively tiny, and instead they could have a swarm of devs (like me) pecking away at AMD compatibility on AI repos if they sold cheap 32GB/48GB cards.

Never said it was ok! Just saying that there are people willing to pay this much, so it costs this much. I'd very much like to buy a 40GB GPU for this to, but at these prices this is not happening - I'd have to turn it into a business to justify this expense, but I just don't feel like it.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#236

Post by brucethemoose2 »

stolsvik wrote: post #228
behnamoh wrote: post #215

Earlier quoted context omitted.

I think the current approach — train 7b models and then do MoE on them — is the future. It’ll still be only runnable on high end customer devices. As for 13b + MoE, I don’t think any customer device could handle that in the next couple years.

I have no formal credentials to say this, but intuitively I feel this is obviously wrong. You couldn’t have taken 50 rats brains and “mixed” them and expected the result to produce new science. For some uninteresting regurgitation, sure. But size - width and depth - seems like an important piece for ability to extract deep understanding of the universe. Also, MoE, as I understand it, will inherently not be able to gl…

The MOE models are essentially trained as a single model. Its not 7 independent models, individually (AFAIK) they are all totally useless without each other.

Its just that each bit picks up different "parts" of the training more strongly, which can be selectively picked at runtime. This is actually kinda analogous to animals, which dont fire every single neuron so frequently like monolithic models do.

The tradeoff, at equivalent quality, is essentially increased VRAM usage for faster, more splittable inference and training, though the exact balance of this tradeoff is an excellent question.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#237

Post by jeanloolz »

leobg wrote: post #141

I love Mistral. It’s crazy what can be done with this small model and 2 hours of fine tuning. Chatbot with function calling? Check. 90 +% accuracy multi label classifier, even when you only have 15 examples for each label? Check. Craaaazy powerful.

Can you point me to a function calling fine tune mistral model? This is the only feature that keeps me from migrating away from openai. I searched a few time but could not find anything in HG

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#238

Post by leobg »

jeanloolz wrote: post #237
leobg wrote: post #141

I love Mistral. It’s crazy what can be done with this small model and 2 hours of fine tuning. Chatbot with function calling? Check. 90 +% accuracy multi label classifier, even when you only have 15 examples for each label? Check. Craaaazy powerful.

Can you point me to a function calling fine tune mistral model? This is the only feature that keeps me from migrating away from openai. I searched a few time but could not find anything in HG

Can’t share the model, since it was trained for a client. I don’t know if any public datasets exist. But Mistral will learn what you throw at it. So if you build a dataset of chat conversations that contains, say, answers in the form of {“answer”:”The answer”, “image”:”Prompt for stable diffusion”}, you’ll get a model that can generate images, and also will know when to use that capability. It’s insane how well that works.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#239

Post by leobg »

leodriesch wrote: post #192
leobg wrote: post #141

I love Mistral. It’s crazy what can be done with this small model and 2 hours of fine tuning. Chatbot with function calling? Check. 90 +% accuracy multi label classifier, even when you only have 15 examples for each label? Check. Craaaazy powerful.

Could you link me to a finetune optimized for function calling? I was looking for one a few weeks ago but did not find any.

See sibling comment.
Post reply on HN