Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mixtral 8x22B

mistral.ai

Post reply on HN
Print view View original

71–80 of 252 posts

Re: Mixtral 8x22B

#71

Post by andai »

CharlieDigital wrote: post #29
apetresc wrote: post #16

Earlier quoted context omitted.

Virtually every announcement of a new model release has some sort of table or graph matching it up against a bunch of other models on various benchmarks, and they're always selected in such a way that the newly-released model dominates along several axes. It turns interpreting the results into an exercise in detecting which models and benchmarks were omitted.

It would make sense, wouldn't it? Just as we've seen rising fuel efficiency, safety, dependability, etc. over the lifecycle of a particular car model. The different teams are learning from each other and pushing boundaries; there's virtually no reason for any of the teams to release a model or product that is somehow inferior to a prior one (unless it had some secondary attribute such as requiring lower end hardware)…

Sibling comment made a good point about benchmarks not being a great indiactor of real world quality. Every time something scores near GPT-4 on benchmarks, I try it out and it ends up being less reliable than GPT-3 within a few minutes of usage.

Re: Mixtral 8x22B

#72

Post by jsemrau »

jjice wrote: post #27

Does anyone have a good layman's explanation of the "Mixture-of-Experts" concept? I think I understand the idea of having "sub-experts", but how do you decide what each specialization is during training? Or is that not how it works at all?

There is some good documentation around mergekit available that actually explains a lot and might be a good place to start.

Re: Mixtral 8x22B

#74

Post by luke-stanley »

I'm confused on the instruction fine-tuning part that is mentioned briefly, in passing. Is there an open weight instruct variant they've released? Or is that only on their platform? Edit: It's on HuggingFace, great, thanks replies!

Re: Mixtral 8x22B

#76

Post by andai »

hlfshell wrote: post #44
jjice wrote: post #27

Does anyone have a good layman's explanation of the "Mixture-of-Experts" concept? I think I understand the idea of having "sub-experts", but how do you decide what each specialization is during training? Or is that not how it works at all?

This is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can't possibly track. During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks. The result of this is that each token has essentially even probability of being routed to one of the sub m…

I heard MoE reduces inference costs. Is that true? Don't all the sub networks need to be kept in RAM the whole time? Or is the idea that it only needs to run compute on a small part of the total network, so it runs faster? (So you complete more requests per minute on same hardware.)

Edit: Apparently each part of the network is on a separate device. Fascinating! That would also explain why the routing network is trained to choose equally between experts.

I imagine that may reduce quality somewhat though? By forcing it to distribute problems equally across all of them, whereas in reality you'd expect task type to conform to the pareto distribution.

Re: Mixtral 8x22B

#77

Post by rgbrgb »

fire_lake wrote: post #54
hlfshell wrote: post #44

Earlier quoted context omitted.

This is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can't possibly track. During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks. The result of this is that each token has essentially even probability of being routed to one of the sub m…

Why do we expect this to perform better? Couldn’t a regular network converge on this structure anyways?

Here's my naive intuition: in general bigger models can store more knowledge but take longer to do inference. MoE provides a way to blend the advantages of having a bigger model (more storage) with the advantages of having smaller models at inference time (faster, less memory required). When you do inference, tokens hit a small layer that is load balancing the experts then activate 1 or 2 experts. So you're storing roughly 8 x 22B "worth" of knowledge without having to run a model that big.

Maybe a real expert can confirm if this is correct :)

Re: Mixtral 8x22B

#79

Post by freedmand »

luke-stanley wrote: post #74

I'm confused on the instruction fine-tuning part that is mentioned briefly, in passing. Is there an open weight instruct variant they've released? Or is that only on their platform? Edit: It's on HuggingFace, great, thanks replies!

I just found this on HuggingFace: https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1

Re: Mixtral 8x22B

#80

Post by ayolisup »

What's the best way to run this on my Macbook Pro?

I've tried LMStudio, but I'm not a fan of the interface compared to OpenAI's. The lack of automatic regeneration every time I edit my input, like on ChatGPT, is quite frustrating. I also gave Ollama a shot, but using the CLI is less convenient.

Ideally, I'd like something that allows me to edit my settings quite granularly, similar to what I can do in OpenLM, with the QoL from the hosted online platforms, particularly the ease of editing my prompts that I use extensively.

Post reply on HN