Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mixtral 8x22B

mistral.ai

Post reply on HN
Print view View original

171–180 of 252 posts

Re: Mixtral 8x22B

#171

Post by MPSimmons »

andai wrote: post #76
hlfshell wrote: post #44

Earlier quoted context omitted.

This is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can't possibly track. During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks. The result of this is that each token has essentially even probability of being routed to one of the sub m…

I heard MoE reduces inference costs. Is that true? Don't all the sub networks need to be kept in RAM the whole time? Or is the idea that it only needs to run compute on a small part of the total network, so it runs faster? (So you complete more requests per minute on same hardware.) Edit: Apparently each part of the network is on a separate device. Fascinating! That would also explain why the routing network is train…

>I heard MoE reduces inference costs

Computational costs, yes. You still take the same amount of time for processing the prompt, but each token created through inference costs less computationally than if you were running it through _all_ layers.

Re: Mixtral 8x22B

#172

Post by samus »

andai wrote: post #76
hlfshell wrote: post #44

Earlier quoted context omitted.

This is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can't possibly track. During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks. The result of this is that each token has essentially even probability of being routed to one of the sub m…

I heard MoE reduces inference costs. Is that true? Don't all the sub networks need to be kept in RAM the whole time? Or is the idea that it only needs to run compute on a small part of the total network, so it runs faster? (So you complete more requests per minute on same hardware.) Edit: Apparently each part of the network is on a separate device. Fascinating! That would also explain why the routing network is train…

It should increase quality since those layers can specialize on subsets of the training data. This means that getting better in one domain won't make the model worse in all the others anymore.

We can't really tell what the router does. There have been experiments where the router in the early blocks was compromised, and quality only suffered moderately. In later layers, as the embeddings pick up more semantic information, it matters more and might approach our naive understanding of the term "expert".

Re: Mixtral 8x22B

#174

Post by moffkalast »

pseudosavant wrote: post #157
mdrzn wrote: post #14

"64K tokens context window" I do wish they had managed to extend it to at least 128K to match the capabilities of GPT-4 Turbo Maybe this limit will become a joke when looking back? Can you imagine reaching a trillion tokens context window in the future, as Sam speculated on Lex's podcast?

FWIW, the 128k context window for GPT-4 is only for input. I believe the output content is still only 4k.

How does that make any sense on a decoder-only architecture?

Re: Mixtral 8x22B

#175

Post by samus »

rgbrgb wrote: post #77
fire_lake wrote: post #54

Earlier quoted context omitted.

Why do we expect this to perform better? Couldn’t a regular network converge on this structure anyways?

Here's my naive intuition: in general bigger models can store more knowledge but take longer to do inference. MoE provides a way to blend the advantages of having a bigger model (more storage) with the advantages of having smaller models at inference time (faster, less memory required). When you do inference, tokens hit a small layer that is load balancing the experts then activate 1 or 2 experts. So you're storing r…

Almost :) the model chooses experts in every block. For a typical 7B with 8 experts there will be 8^32=2^96 paths through the whole model.

Re: Mixtral 8x22B

#176

Post by pradn »

wongarsu wrote: post #110
paxys wrote: post #50

Earlier quoted context omitted.

We are going to quickly reach the point where most of these AI startups (which do nothing but provide thin wrappers on top of public LLMs) aren't going to be needed at all. The differentiation will need to come from the value of the end product put in front of customers, not the AI backend.

The same happened to image recognition. We have great algorithms for many years now. You can't make a company out of having the best image recognition algorithm, but you absolutely can make a company out of a device that spots defects in the paintjob in a car factory, or that spots concrete cracks in the tunnel segments used by a tunnel boring machine, or by building a wildlife camera that counts wildlife and exports…

The key thing is AI models are services not products. The real world changes, so you have to change your model. Same goes for new training data (examples, yes/no labels, feedback from production use), updating biases (compliance, changing societal mores). And running models in a highly-available way is also expertise. Not every company wants to be in the ML-ops business.

Re: Mixtral 8x22B

#178

Post by chown »

ayolisup wrote: post #80

What's the best way to run this on my Macbook Pro? I've tried LMStudio, but I'm not a fan of the interface compared to OpenAI's. The lack of automatic regeneration every time I edit my input, like on ChatGPT, is quite frustrating. I also gave Ollama a shot, but using the CLI is less convenient. Ideally, I'd like something that allows me to edit my settings quite granularly, similar to what I can do in OpenLM, with th…

You can try Msty as well. I am the author.

https://msty.app

Re: Mixtral 8x22B

#179

Post by Quarrel »

huevosabio wrote: post #88
jjice wrote: post #27

Does anyone have a good layman's explanation of the "Mixture-of-Experts" concept? I think I understand the idea of having "sub-experts", but how do you decide what each specialization is during training? Or is that not how it works at all?

Ignore the "experts" part, it misleads a lot of people [0]. There is no explicit specialization in the most popular setups, it is achieved implicitly through training. In short: MoEs add multiple MLP sublayers and a routing mechanism after each attention sublayer and let the training procedure learn the MLP parameters and the routing parameters. In a longer, but still rough, form... How these transformers work is rou…

So it is somewhat like a classic random forest or maybe bagging, where you're trying to stop overfitting, but you're also trying to train that top layer to know who could be the "experts" given the current inputs so that you're minimising the number of multiple MLP sublayers called during inference?

Re: Mixtral 8x22B

#180

Post by mcbuilder »

ayolisup wrote: post #80

What's the best way to run this on my Macbook Pro? I've tried LMStudio, but I'm not a fan of the interface compared to OpenAI's. The lack of automatic regeneration every time I edit my input, like on ChatGPT, is quite frustrating. I also gave Ollama a shot, but using the CLI is less convenient. Ideally, I'd like something that allows me to edit my settings quite granularly, similar to what I can do in OpenLM, with th…

openrouter.ai is a fantastic idea if you don't want to self host
Post reply on HN