Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mixtral 8x22B

mistral.ai

Post reply on HN
Print view View original

191–200 of 252 posts

Re: Mixtral 8x22B

#191

Post by kristopolous »

sva_ wrote: post #99
kristopolous wrote: post #62

Earlier quoted context omitted.

Everything is soldered in these days. It's complete garbage. And most of the other vendors just copy Apple so even things like Lenovo have the same problems. The current state of laptops is such trash

Plenty of laptops still have SO-DIMM, such as EliteBook for example. People need to vote with their wallet, and not buy stuff that goes against their principles.

There needs to be more fidelity than "vote with wallet". Let's say I decided to not purchase your product. Why?

The question remains unanswered. Perhaps I didn't see it for sale or Bob in accounting just got one and I didn't want to look like I was copying Bob.

Even at scale this doesn't work. Let's say Lenovo switches to making all of their laptops hot pink with bedazzled rhinestone butterflies and sales plummet. You could argue it was the wrong pink or that the butterflies didn't shimmer enough ... any hypothesis you wish.

The market provides an extremely low information poor signal that really doesn't suggest any course of action.

If we really want something better, there needs to be more fruitful and meaningful communication lines. I've come up with various ideas over the years but haven't really implemented them.

Re: Mixtral 8x22B

#192

Post by londons_explore »

ru552 wrote: post #9
imjonse wrote: post #3

Great to see such free to use and self-hostable models, but it's said that open now means only that. One cannot replicate this model without access to the training data.

There's a large amount of liability in disclosing your training data.

I expect we'll see some AI companies in the future throwing away the training dataset. Maybe some have already.

During a court case, the other side can demand discovery over your training dataset, for example to see if it contains a particular copyrighted work.

But if you've already deleted the dataset, you're far more likely to win any case against you that hinges on what was in the dataset if the plaintiff can't even prove their work was included.

And you can argue that the dataset was very expensive to store (which is true), and therefore deleted shortly after training was complete. You have no obligation to keep something for the benefit of potential future plaintiffs you aren't even aware of yet.

Re: Mixtral 8x22B

#193

Post by huevosabio »

Quarrel wrote: post #179
huevosabio wrote: post #88

Earlier quoted context omitted.

Ignore the "experts" part, it misleads a lot of people [0]. There is no explicit specialization in the most popular setups, it is achieved implicitly through training. In short: MoEs add multiple MLP sublayers and a routing mechanism after each attention sublayer and let the training procedure learn the MLP parameters and the routing parameters. In a longer, but still rough, form... How these transformers work is rou…

So it is somewhat like a classic random forest or maybe bagging, where you're trying to stop overfitting, but you're also trying to train that top layer to know who could be the "experts" given the current inputs so that you're minimising the number of multiple MLP sublayers called during inference?

Yea, it's very much bagging + top layer (router) for the importance score!

Re: Mixtral 8x22B

#194

Post by HeatrayEnjoyer »

wongarsu wrote: post #110
paxys wrote: post #50

Earlier quoted context omitted.

We are going to quickly reach the point where most of these AI startups (which do nothing but provide thin wrappers on top of public LLMs) aren't going to be needed at all. The differentiation will need to come from the value of the end product put in front of customers, not the AI backend.

The same happened to image recognition. We have great algorithms for many years now. You can't make a company out of having the best image recognition algorithm, but you absolutely can make a company out of a device that spots defects in the paintjob in a car factory, or that spots concrete cracks in the tunnel segments used by a tunnel boring machine, or by building a wildlife camera that counts wildlife and exports…

The dynamic does seem to be different with the newer systems. Larger more general systems are better than small specialized models.

GPT-4 is SOTA at OCR and sentiment classification, for example.

Re: Mixtral 8x22B

#195

Post by jerpint »

jjice wrote: post #27

Does anyone have a good layman's explanation of the "Mixture-of-Experts" concept? I think I understand the idea of having "sub-experts", but how do you decide what each specialization is during training? Or is that not how it works at all?

The simplest way to think about it is a form of dropout but instead of dropping weights, you drop an entire path of the network

Re: Mixtral 8x22B

#196

Post by orra »

Is this release a pleasant surprise? Mistral weakened their commitment to open source when they partnered with Microsoft.

It's nice they're using some of the money from their commercial and proprietary models, to improve the state of the art for open source (open weights) models.

Re: Mixtral 8x22B

#197

Post by DougBTX »

huevosabio wrote: post #88
jjice wrote: post #27

Does anyone have a good layman's explanation of the "Mixture-of-Experts" concept? I think I understand the idea of having "sub-experts", but how do you decide what each specialization is during training? Or is that not how it works at all?

Ignore the "experts" part, it misleads a lot of people [0]. There is no explicit specialization in the most popular setups, it is achieved implicitly through training. In short: MoEs add multiple MLP sublayers and a routing mechanism after each attention sublayer and let the training procedure learn the MLP parameters and the routing parameters. In a longer, but still rough, form... How these transformers work is rou…

Would this be a reasonable explanation?

> MLPs are universal function approximators, but these models are big enough that it is better to train many small functions rather than a single unified function. MoE is a mechanism to force different parts of the model to learn distinct functions.

Re: Mixtral 8x22B

#198

Post by samus »

andai wrote: post #100
hlfshell wrote: post #44

Earlier quoted context omitted.

This is a bit of a misnomer. Each expert is a sub network that specializes in sub understanding we can't possibly track. During training a routing network is punished if it does not evenly distribute training tokens to the correct experts. This prevents any one or two networks from becoming the primary networks. The result of this is that each token has essentially even probability of being routed to one of the sub m…

Has anyone tried MoE at smaller scales? e.g. a 7B model that's made of a bunch of smaller ones? I guess that would be 8x1B. Or would that make each expert too small to be useful? TinyLlama is 1B and it's almost useful! I guess 8x1B would be Mixture of TinyLLaMAs...

There is Qwen1.5-MoE-A2.7B, which was made by upcycling the weights of Qwen1.5-1.8B, splitting it and finetuning it.

Re: Mixtral 8x22B

#199

Post by bevekspldnw »

imjonse wrote: post #149
Lacerda69 wrote: post #57

Earlier quoted context omitted.

It's open enough for all practical purposes IMO.

As open as an executable binary that you are allowed to download and use for free.

In the age of SaaS I’ll take it. It’s not like I have a few million dollars to pay for training even if I had all the code and data.

Re: Mixtral 8x22B

#200

Post by samus »

DougBTX wrote: post #197
huevosabio wrote: post #88

Earlier quoted context omitted.

Ignore the "experts" part, it misleads a lot of people [0]. There is no explicit specialization in the most popular setups, it is achieved implicitly through training. In short: MoEs add multiple MLP sublayers and a routing mechanism after each attention sublayer and let the training procedure learn the MLP parameters and the routing parameters. In a longer, but still rough, form... How these transformers work is rou…

Would this be a reasonable explanation? > MLPs are universal function approximators, but these models are big enough that it is better to train many small functions rather than a single unified function. MoE is a mechanism to force different parts of the model to learn distinct functions.

It misses the crucial detail that every transformer layer chooses the experts independently from the others. Of course they still indirectly influence each other since each layer processes the output of the previous one.
Post reply on HN