Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral "Mixtral" 8x7B 32k model [magnet]

twitter.com

Post reply on HN
Print view View original

191–200 of 255 posts

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#191

Post by GaggiX »

BryanLegend wrote: post #98

Andrej Karpathy's take: New open weights LLM from @MistralAI params.json: - hidden_dim / dim = 14336/4096 => 3.5X MLP expand - n_heads / n_kv_heads = 32/8 => 4X multiquery - "moe" => mixture of experts 8X top 2 Likely related code: https://github.com/mistralai/megablocks-public Oddly absent: an over-rehearsed professional release video talking about a revolution in AI. If people are wondering why there is so much AI…

>-hidden_dim / dim = 14336/4096 => 3.5X MLP expand

>- n_heads / n_kv_heads = 32/8 => 4X

These two are exactly the same as the old Mistral-7B

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#192

Post by leodriesch »

leobg wrote: post #141

I love Mistral. It’s crazy what can be done with this small model and 2 hours of fine tuning. Chatbot with function calling? Check. 90 +% accuracy multi label classifier, even when you only have 15 examples for each label? Check. Craaaazy powerful.

Could you link me to a finetune optimized for function calling? I was looking for one a few weeks ago but did not find any.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#193

Post by nicolas03 »

brucethemoose2 wrote: post #189
faldore wrote: post #50

Earlier quoted context omitted.

at 4 bits you could run it on a 3090 right?

Its crazy how the 3090 is such a ubiquitous local llm card these days. I despise Nvidia on linux... And yet I ended up with a 3090. How are AMD/Intel totally missing this boat?

LMAO SAME. I hate nvidia yet got a used 3090 for $600. I’ve been biting my nails hoping china dosent resort to 3090’s, because I really want to buy another and I’m not paying more than 600.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#194

Post by jbarrow »

crakenzak wrote: post #182
BryanLegend wrote: post #98

Andrej Karpathy's take: New open weights LLM from @MistralAI params.json: - hidden_dim / dim = 14336/4096 => 3.5X MLP expand - n_heads / n_kv_heads = 32/8 => 4X multiquery - "moe" => mixture of experts 8X top 2 Likely related code: https://github.com/mistralai/megablocks-public Oddly absent: an over-rehearsed professional release video talking about a revolution in AI. If people are wondering why there is so much AI…

> it's because the biggest deep learning conference (NeurIPS) is next week. Can we expect some big announcements (new architectures, models, etc) at the conference from different companies? Sorry, not too familiar what the culture for research conferences is.

Typically not. Google as an example: the transformer paper (Vaswani et al., 2017) was arxiv'd in June of 2017, and NeurIPS (the conference in which it was published) was in December of that year; BERT (Devlin et al., 2019) was similarly arxiv'd before publication.

Recent announcements from companies tend to be even more divorced from conference dates, as they release anemic "Technical Reports" that largely wouldn't pass muster in a peer review.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#198

Post by nullc »

rishabhjain1198 wrote: post #173
moffkalast wrote: post #69

Earlier quoted context omitted.

It means it's 8 7B models in a trench coat in a sense, it runs as fast as a 14B (2 experts at a time apparently) but takes up as much memory as a 40B model (70% * 8 * 7B). There is some process trained into it that chooses which experts to use based on the question posed. GPT 4 is allegedly based on the same architecture, but at 8*222B.

In a MoE model with experts_per_token = 2 and each expert having 7B params, after picking the experts it should run as fast as the slowest 7B expert, not a comparable 14B model.

Only assuming it's able to hide the faster one in free parallelism.

Re: Mistral "Mixtral" 8x7B 32k model [magnet]

#200

Post by screye »

brucethemoose2 wrote: post #57

In other llm news, Mistral/Yi finetunes trained with a new (still undocumented) technique called "neural alignment" are blasting other models in the HF leaderboard. The 7B is "beating" most 70Bs. The 34B in testing seems... Very good: https://huggingface.co/fblgit/una-xaberius-34b-v1beta https://huggingface.co/fblgit/una-cybertron-7b-v2-bf16 I mention this because it could theoretically be applied to Mistral Moe. If…

Yeah, and Mistral doesn't particularly care about lobotomizing the model with 'safety-training'. So it can achieve much better performance per-parameter than anthropic/google/OpenAI while being more steerable as well.
Post reply on HN