Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

infoq.com

Post reply on HN
Print view View original

21–30 of 78 posts

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#21

Post by gradys »

buildbot wrote: post #7

If the model weights aren’t open source, the model isn’t open source in my opinion. It’s not like a switch transformer is that much more complicated to slap into pytorch than any other new research. Also in my personal opinion, while yes this model performs great, it’s ridiculously parameter inefficient. 1.6T parameters is 3.12TB using bfloat 16. I’d rather take 7x longer to train and have a model I can actually put…

Mixture of experts architectures like this are a specific design decision to increase the parameter count but use and update those parameters sparsely. Sure, that design decision doesn't fit all scenarios, but it fits some, and it has its own advantages, like faster training time.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#22

Post by whimsicalism »

sp332 wrote: post #13
amelius wrote: post #12

Since this type of model can be used for the task of "question answering", would it be possible to create a search engine with this type of model?

You could make something like an "expert system" with it. But since it doesn't seem to link back to the source, you'd never know if it was giving you a real answer or just making something up.

A bit like people

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#24

Post by gradys »

huslage wrote: post #16

Can someone translate this post into english for those of us not imbued with AI knowledge?

One key idea here is to use a very large number of parameters (model weights), but only use some subset of the parameters on each example. The parameters are divided up into blocks called "experts", and then some subset of experts are used on any given input. Which subset is used is chosen by the model itself in a data-dependent manner. This can be thought of as letting the model specialize different experts to handle different situations.

The advantage, as they show, is that the model can train to a given level of performance much faster with a fixed amount of computing power compared to an architecture that uses all parameters on every step. This might be because it allows you to have a very large number of parameters that can store a lot more specialized information without incurring as much of a computational cost. Of course the downside is that you end up with a very large model that literally won't fit in a lot of environments.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#25

Post by lifthrasiir »

schoen wrote: post #10
buildbot wrote: post #8

Earlier quoted context omitted.

If I did the math right it would be 3.12TB of weights, maybe they are trying to upload it to gdrive still. (/s, probably)

They released more data than that for their Google Books n-grams datasets: https://storage.googleapis.com/books/ngrams/books/datasetsv3... (I don't remember exactly how much it is, but I remember that the old version was already in the terabytes.)

Another example of Google giving much data away is 50 trillion digits of pi [1], which contains about 42 TB of data (decimal and hexadecimal combined).

[1] https://storage.googleapis.com/pi50t/index.html

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#26

Post by jariel »

So, almost every possible ngram ever used is a 'input'?

Can someone describe what the lexical reality of 3.5T inputs actually means?

I feel like this is 'Deep Memorization' instead of 'Deep Learning'.

Like a Doctor who passes everything merely by memorizing the textbook with absolutely no ability beyond that.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#28

Post by dekhn »

buildbot wrote: post #8
strin wrote: post #2

lol they didn’t open source the model weights

If I did the math right it would be 3.12TB of weights, maybe they are trying to upload it to gdrive still. (/s, probably)

Google Cloud Storage. The files could be dumped as tfrecord in a bucket with "requester pays". So anybody could reproduce it using the open source code, by paying for the costs incurred to move the data from GCS to the training nodes.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#29

Post by neximo64 »

airocker wrote: post #23

Would GDPR or other regulation apply to the data that these model is trained with? Is it not a risk that the model will record some private information in someone's email?

Maybe host and use it in a non GDPR environment, it's always going to be ambiguous and risky.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#30

Post by vladf »

thesz wrote: post #18
amelius wrote: post #9

Earlier quoted context omitted.

> it’s ridiculously parameter inefficient How do you know that? Perhaps with this methodology you really need those 3.12TB to reach comparable performance?

I beg to disagree. [1] provides one with a whole-data-set training method (ADMM, one of such methods). Page 8 contains figure 2(b) - accuracy of training after specified amount of time. Note that ADMM start where stochastic gradient stops. [1] https://arxiv.org/pdf/1605.02026.pdf At [2] I tried to apply logistic regression trained using reweighted least squares algorithm on the same Higgs boson data set. I've got the…

> they do not quite care about parameter efficiency.

Google Research is pretty big, I used to think like you did but I think it's mostly b/c DeepMind just hogs all the spotlight.

Check out PRESS [0] for example.

[0]: https://research.google/pubs/pub46141

Post reply on HN