Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Implementation of mixture of experts language model in a single file of PyTorch

github.com

Post reply on HN
Print view View original

11–15 of 15 posts

Re: Implementation of mixture of experts language model in a single file of PyTorch

#11

Post by zingelshuher »

avisoori1x wrote: post #6
radarsat1 wrote: post #3

Earlier quoted context omitted.

Adding scaled unit gaussian noise to the logits noise = torch.randn_like(logits)*F.softplus(noise_logits) noisy_logits = logits + noise Question, if you changed this Gaussian normal for Gumbel noise you would get something like Gumbel softmax, right? I'm curious why not use it? Isn't it a usual way to implement differentiable discrete selection? My curiosity is about the effectiveness of Gumbel softmax since I have h…

This is a good point. I'm yet to try it as I've kind of let this project sit for a couple of months and only getting back to it. I went with this because it's simpler but I'm not sure simpler is necessarily better in this case.

Question, have you seen the improvement after adding the noise? I mean in practice. Asking because intuition sometimes doesn't work.

Re: Implementation of mixture of experts language model in a single file of PyTorch

#12

Post by radarsat1 »

avisoori1x wrote: post #6
radarsat1 wrote: post #3

Earlier quoted context omitted.

Adding scaled unit gaussian noise to the logits noise = torch.randn_like(logits)*F.softplus(noise_logits) noisy_logits = logits + noise Question, if you changed this Gaussian normal for Gumbel noise you would get something like Gumbel softmax, right? I'm curious why not use it? Isn't it a usual way to implement differentiable discrete selection? My curiosity is about the effectiveness of Gumbel softmax since I have h…

This is a good point. I'm yet to try it as I've kind of let this project sit for a couple of months and only getting back to it. I went with this because it's simpler but I'm not sure simpler is necessarily better in this case.

Ah ok, I was wondering if there was some theory here that I wasn't aware of but if it's just experimentation no problem ;) good to know in any case!

I find it a bit difficult to find resources describing the properties of various options for this topic of discrete choices and clustering, apart from a few papers & blogs describing the idea.

Re: Implementation of mixture of experts language model in a single file of PyTorch

#13

Post by haeggee »

very cool work! we did something similar in the context of the Swiss AI initiative (https://www.swiss-ai.org/) here: https://github.com/swiss-ai/MoE. The implementation is as simple and fast as nanoGPT and works with our modular llm-baselines codebase (https://github.com/epfml/llm-baselines) for experimenting with transformers and different datasets :)

Re: Implementation of mixture of experts language model in a single file of PyTorch

#14

Post by avisoori1x »

zingelshuher wrote: post #11
avisoori1x wrote: post #6

Earlier quoted context omitted.

This is a good point. I'm yet to try it as I've kind of let this project sit for a couple of months and only getting back to it. I went with this because it's simpler but I'm not sure simpler is necessarily better in this case.

Question, have you seen the improvement after adding the noise? I mean in practice. Asking because intuition sometimes doesn't work.

Quite honestly not in my experiments. I wanted to do some Bayesian hyperparameter optimization with some discretized options like noise/no-noise and n_expert/top_k but haven't been able to find the time or free time in one of our GPU clusters. I plan on using perplexity as this is not yet instruction fine tuned.

Re: Implementation of mixture of experts language model in a single file of PyTorch

#15

Post by avisoori1x »

haeggee wrote: post #13

very cool work! we did something similar in the context of the Swiss AI initiative ( https://www.swiss-ai.org/ ) here: https://github.com/swiss-ai/MoE . The implementation is as simple and fast as nanoGPT and works with our modular llm-baselines codebase ( https://github.com/epfml/llm-baselines ) for experimenting with transformers and different datasets :)

This is awesome! Thanks for sharing. I'll definitely check this out.
Post reply on HN