71–78 of 78 posts
Would GDPR or other regulation apply to the data that these model is trained with? Is it not a risk that the model will record some private information in someone's email?
Earlier quoted context omitted.
Mixture of experts architectures like this are a specific design decision to increase the parameter count but use and update those parameters sparsely. Sure, that design decision doesn't fit all scenarios, but it fits some, and it has its own advantages, like faster training time.
But do you see any practical scenario where you have to keep 3TB of parameters in accelerator's RAM at inference time? The accuracy would have to be significantly higher that any alternative to justify monopolizing that many hardware resources.
This kind of model could be used as the teacher in a distillation setup too though. Then faster training of the teacher is actually a huge benefit since it speeds up model development iteration cycles.
But even if it weren't practical to use in production in any sense, I'd argue there's value in doing the basic research of exploring design space of architectures in this way. This came out of a research team at Google. It may inspire and inform smaller, more practical architectures.
There is something fundamentally wrong with these models. The brain "works" because it's evolved structure matches or reflects reality. It is not about having billions of neurons, but about to have the right structure which matches the environment. My favourite example is how butterflies evolve pictures of eyes on its wings to scare predators, having literally no idea about existence of other creatures. It has been e…
Earlier quoted context omitted.
But do you see any practical scenario where you have to keep 3TB of parameters in accelerator's RAM at inference time? The accuracy would have to be significantly higher that any alternative to justify monopolizing that many hardware resources.
Yep, it would definitely be difficult to justify running it in production. Accuracy would need to be higher as you said or it would need to be applicable to more tasks such that you can take other models out of production. This kind of model could be used as the teacher in a distillation setup too though. Then faster training of the teacher is actually a huge benefit since it speeds up model development iteration cyc…
Part of the justification is the MoE sparsity by design means only a small part of the model will be activated by a given query. Don't think of it as a single giant 1t model, think of it as 50 small models which happen to share a glue layer at the input. So at deployment, you could, for example, keep the gating-layer in RAM and only pull the necessary sub-model off disk as necessary. Or you could shard the sub-models over 51 GPUs and feed the master gating-layer+GPU lots of queries, and each query will be dispatched to a different expert+GPU pair. This could easily be competitive with running a lot of dense models in parallel trying to keep up with the same load.
Earlier quoted context omitted.
Another naive question: why is this better than creating a separate, smaller model for each expert?
The common argument I've heard: because then you would have to decide how many experts models are required, train and evaluate them separately, and overall make your architecture dependent on this choice. If your expert is wrong and miscalculates how many models are required then your entire architecture is also likely to be wrong (humans, am I right?). Researchers at Google's scale prefer a single model where you th…
Earlier quoted context omitted.
> they do not quite care about parameter efficiency. Google Research is pretty big, I used to think like you did but I think it's mostly b/c DeepMind just hogs all the spotlight. Check out PRESS [0] for example. [0]: https://research.google/pubs/pub46141
Thank you! I skimmed over the abstract and will read the paper later, it seems interesting. But you gave me another point to support my view: PRESS uses stochastic gradient, not second-order method like IRLS.
For instance, on modern modelling problems with non-linearities and change points, it's a lot less easy to do something like IRLS in an end-to-end system, but interesting as a research direction.
Earlier quoted context omitted.
Magic algorithms don’t exist. That’s the true reality of AI. I also felt the same thing and became dismayed as an undergrad at what was going on in computation having also studied neurobiology. But as it turns out, a shit ton of data with some stats can get you very, very far.
I didn't mean 'magic' (I know it's not that) - I just meant to imply that I think of AI as a 'function' not a 'lookup'. Inputs -> Outputs not Search -> Response Like if you train AI on a small dataset, it feels like what it's doing afterwords is a 'function' or 'algorithm' using what is in the end some arcane algebra. But if you train on all the data in the world, with a trillion parameters ... well ... I kind of fee…
Earlier quoted context omitted.
Thank you! I skimmed over the abstract and will read the paper later, it seems interesting. But you gave me another point to support my view: PRESS uses stochastic gradient, not second-order method like IRLS.
I agree, I was really only proposing PRESS for the "parameter efficiency" part of your comment. It'd be interesting to see some modern takes on IRLS. I think generally this goes against the grain of the Cheap Gradient Principle which is why you see less of it (edit: eh, I think Fisher scoring can be cast in this light). For instance, on modern modelling problems with non-linearities and change points, it's a lot less…
The very SGD thing was developed because it was the only way to train something like neural network with small memory and, more importantly, in reasonable time. Multiplying of training time by N (number of parameter) meant having good result in a year, not in a day.
And today we have large batch training with complex synchronization systems to speed up training even more. Which bring us closer to the whole-dataset training and, I guess, second-order optimization as well.