Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

infoq.com

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#41

Post by jariel »

visarga wrote: post #39
jariel wrote: post #26

So, almost every possible ngram ever used is a 'input'? Can someone describe what the lexical reality of 3.5T inputs actually means? I feel like this is 'Deep Memorization' instead of 'Deep Learning'. Like a Doctor who passes everything merely by memorizing the textbook with absolutely no ability beyond that.

Even so, memorisation implies word by word information retrieval and interpolation, it's not a hash table.

Yes, I mean, that's cool, but I'm just thinking that this isn't quite the AI we were thinking about before.

It's like a 'new form of storage and lookup' as opposed to the kind of 'magic algorithm' we usually think of when we think of AI. Or maybe that's just me.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#42

Post by azinman2 »

jariel wrote: post #41
visarga wrote: post #39

Earlier quoted context omitted.

Even so, memorisation implies word by word information retrieval and interpolation, it's not a hash table.

Yes, I mean, that's cool, but I'm just thinking that this isn't quite the AI we were thinking about before. It's like a 'new form of storage and lookup' as opposed to the kind of 'magic algorithm' we usually think of when we think of AI. Or maybe that's just me.

Magic algorithms don’t exist. That’s the true reality of AI. I also felt the same thing and became dismayed as an undergrad at what was going on in computation having also studied neurobiology. But as it turns out, a shit ton of data with some stats can get you very, very far.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#43

Post by stingraycharles »

gradys wrote: post #24
huslage wrote: post #16

Can someone translate this post into english for those of us not imbued with AI knowledge?

One key idea here is to use a very large number of parameters (model weights), but only use some subset of the parameters on each example. The parameters are divided up into blocks called "experts", and then some subset of experts are used on any given input. Which subset is used is chosen by the model itself in a data-dependent manner. This can be thought of as letting the model specialize different experts to handl…

Another naive question: why is this better than creating a separate, smaller model for each expert?

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#44

Post by summerlight »

buildbot wrote: post #7

If the model weights aren’t open source, the model isn’t open source in my opinion. It’s not like a switch transformer is that much more complicated to slap into pytorch than any other new research. Also in my personal opinion, while yes this model performs great, it’s ridiculously parameter inefficient. 1.6T parameters is 3.12TB using bfloat 16. I’d rather take 7x longer to train and have a model I can actually put…

Google's legal team would be freaking out if any of its engineers propose open sourcing model weights. This is a mine field full of legal and privacy headaches and I guess it is infeasible to audit 1.6T parameters anyway...

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#45

Post by jariel »

azinman2 wrote: post #42
jariel wrote: post #41

Earlier quoted context omitted.

Yes, I mean, that's cool, but I'm just thinking that this isn't quite the AI we were thinking about before. It's like a 'new form of storage and lookup' as opposed to the kind of 'magic algorithm' we usually think of when we think of AI. Or maybe that's just me.

Magic algorithms don’t exist. That’s the true reality of AI. I also felt the same thing and became dismayed as an undergrad at what was going on in computation having also studied neurobiology. But as it turns out, a shit ton of data with some stats can get you very, very far.

I didn't mean 'magic' (I know it's not that) - I just meant to imply that I think of AI as a 'function' not a 'lookup'.

Inputs -> Outputs not Search -> Response

Like if you train AI on a small dataset, it feels like what it's doing afterwords is a 'function' or 'algorithm' using what is in the end some arcane algebra.

But if you train on all the data in the world, with a trillion parameters ... well ... I kind of feel that 'all the data' is in an AI-style datastructure, that we are 'querying' with AI.

But that's just an observers abstraction.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#46

Post by [deleted] »

jariel wrote: post #41
visarga wrote: post #39

Earlier quoted context omitted.

Even so, memorisation implies word by word information retrieval and interpolation, it's not a hash table.

Yes, I mean, that's cool, but I'm just thinking that this isn't quite the AI we were thinking about before. It's like a 'new form of storage and lookup' as opposed to the kind of 'magic algorithm' we usually think of when we think of AI. Or maybe that's just me.

[deleted]

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#47

Post by p1esk »

stingraycharles wrote: post #43
gradys wrote: post #24

Earlier quoted context omitted.

One key idea here is to use a very large number of parameters (model weights), but only use some subset of the parameters on each example. The parameters are divided up into blocks called "experts", and then some subset of experts are used on any given input. Which subset is used is chosen by the model itself in a data-dependent manner. This can be thought of as letting the model specialize different experts to handl…

Another naive question: why is this better than creating a separate, smaller model for each expert?

Yes, you can think of this as a collection of small models, with another model choosing which smaller model to use for each input.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#48

Post by probably_wrong »

stingraycharles wrote: post #43
gradys wrote: post #24

Earlier quoted context omitted.

One key idea here is to use a very large number of parameters (model weights), but only use some subset of the parameters on each example. The parameters are divided up into blocks called "experts", and then some subset of experts are used on any given input. Which subset is used is chosen by the model itself in a data-dependent manner. This can be thought of as letting the model specialize different experts to handl…

Another naive question: why is this better than creating a separate, smaller model for each expert?

The common argument I've heard: because then you would have to decide how many experts models are required, train and evaluate them separately, and overall make your architecture dependent on this choice. If your expert is wrong and miscalculates how many models are required then your entire architecture is also likely to be wrong (humans, am I right?).

Researchers at Google's scale prefer a single model where you throw all your data in a single bin and get perfect performance out, no tweaking and no pesky humans required.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#49

Post by feanaro »

azinman2 wrote: post #42
jariel wrote: post #41

Earlier quoted context omitted.

Yes, I mean, that's cool, but I'm just thinking that this isn't quite the AI we were thinking about before. It's like a 'new form of storage and lookup' as opposed to the kind of 'magic algorithm' we usually think of when we think of AI. Or maybe that's just me.

Magic algorithms don’t exist. That’s the true reality of AI. I also felt the same thing and became dismayed as an undergrad at what was going on in computation having also studied neurobiology. But as it turns out, a shit ton of data with some stats can get you very, very far.

Magic algorithms (or perhaps magic machines) do exist, though. See: humans.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#50

Post by PartiallyTyped »

amelius wrote: post #12

Since this type of model can be used for the task of "question answering", would it be possible to create a search engine with this type of model?

Google already uses a combination of NLP + PageRank to serve search queries [1].

[1] https://en.wikipedia.org/wiki/RankBrain

Post reply on HN