Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

infoq.com

Post reply on HN
Print view View original

51–60 of 78 posts

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#51

Post by Closi »

summerlight wrote: post #44
buildbot wrote: post #7

If the model weights aren’t open source, the model isn’t open source in my opinion. It’s not like a switch transformer is that much more complicated to slap into pytorch than any other new research. Also in my personal opinion, while yes this model performs great, it’s ridiculously parameter inefficient. 1.6T parameters is 3.12TB using bfloat 16. I’d rather take 7x longer to train and have a model I can actually put…

Google's legal team would be freaking out if any of its engineers propose open sourcing model weights. This is a mine field full of legal and privacy headaches and I guess it is infeasible to audit 1.6T parameters anyway...

This is a really good point, and also depends on what the training data is - although arguably if they are using this model to power their services and it can leak personal data that might be an issue in it's own right.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#52

Post by tjbiddle »

exikyut wrote: post #35
shepherdjerred wrote: post #34

Earlier quoted context omitted.

Weights are just numbers (probably floats?), right? This model has 3.12TB of floats??? That's insane. How do you load that into memory for inferencing?

Use x1e.32xlarge on AWS with 3TB of RAM. Just $12,742/mo - https://calculator.aws/#/estimate?id=7428fa81192c57087ac8cdf... Alternatively order something like the HP Z8 with 3TB RAM configured, which is only $75k - https://zworkstations.com/configurations/2040422/ It's interesting. It would take ~six years for the Z8 to break even compared to AWS, but traffic into and out of the machine would be $0, and I don't think…

When you're spending that kind of money on a machine, there's no way you're paying retail price. Sales reps would give you a significant discount.

Also - think you meant 6 months, not 6 years anyhow :)

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#53

Post by robkop »

summerlight wrote: post #44
buildbot wrote: post #7

If the model weights aren’t open source, the model isn’t open source in my opinion. It’s not like a switch transformer is that much more complicated to slap into pytorch than any other new research. Also in my personal opinion, while yes this model performs great, it’s ridiculously parameter inefficient. 1.6T parameters is 3.12TB using bfloat 16. I’d rather take 7x longer to train and have a model I can actually put…

Google's legal team would be freaking out if any of its engineers propose open sourcing model weights. This is a mine field full of legal and privacy headaches and I guess it is infeasible to audit 1.6T parameters anyway...

I have to say that I really doubt this as they've released model weights for their several last few models that have been of nearly this size (and have been trained on the same dataset).

See https://github.com/google-research/text-to-text-transfer-tra...

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#54

Post by ogrisel »

gradys wrote: post #21
buildbot wrote: post #7

If the model weights aren’t open source, the model isn’t open source in my opinion. It’s not like a switch transformer is that much more complicated to slap into pytorch than any other new research. Also in my personal opinion, while yes this model performs great, it’s ridiculously parameter inefficient. 1.6T parameters is 3.12TB using bfloat 16. I’d rather take 7x longer to train and have a model I can actually put…

Mixture of experts architectures like this are a specific design decision to increase the parameter count but use and update those parameters sparsely. Sure, that design decision doesn't fit all scenarios, but it fits some, and it has its own advantages, like faster training time.

But do you see any practical scenario where you have to keep 3TB of parameters in accelerator's RAM at inference time?

The accuracy would have to be significantly higher that any alternative to justify monopolizing that many hardware resources.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#55

Post by mlajtos »

feanaro wrote: post #49
azinman2 wrote: post #42

Earlier quoted context omitted.

Magic algorithms don’t exist. That’s the true reality of AI. I also felt the same thing and became dismayed as an undergrad at what was going on in computation having also studied neurobiology. But as it turns out, a shit ton of data with some stats can get you very, very far.

Magic algorithms (or perhaps magic machines) do exist, though. See: humans.

We have ~150 trillion synapses. If every synapse was equivalent to just one parameter in ANN, then we could be just magic memoization machines.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#56

Post by qayxc »

whimsicalism wrote: post #22
sp332 wrote: post #13

Earlier quoted context omitted.

You could make something like an "expert system" with it. But since it doesn't seem to link back to the source, you'd never know if it was giving you a real answer or just making something up.

A bit like people

Not exactly. People can be asked to provide their sources or reasoning, which an informed user can easily check and verify.

This is not the case with black-box models. They only point to an answer/give a reply without any justification or reasoning behind it. This is actually a very severe problem with black-box models: ultimately they cannot be trusted, because it's very hard to verify whether the learned objective function matches the intended objective function (this is called the alignment problem [1].)

Optimisers tend to produce Clever Hans instances whenever they can, because it's the cheapest and therefore most optimal solution. Even if this becomes obvious from failure cases (e.g. common misclassification in image recognition systems), it's still not obvious which clues the system used that lead to the misclassification.

This is in contrast to a person, who an be queried as to why and how they arrived at their conclusion.

[1] https://bdtechtalks.com/2021/01/18/ai-alignment-problem-bria...

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#57

Post by qayxc »

mlajtos wrote: post #55
feanaro wrote: post #49

Earlier quoted context omitted.

Magic algorithms (or perhaps magic machines) do exist, though. See: humans.

We have ~150 trillion synapses. If every synapse was equivalent to just one parameter in ANN, then we could be just magic memoization machines.

> If every synapse was equivalent to just one parameter in ANN, then we could be just magic memoization machines.

It's a poor comparison, though, since neurons and synapses aren't the same as parameters in a computational network. It's the same trap that news outlets routinely fall into when citing "storage capacity of the brain" in TBs and other such nonsense.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#58

Post by thesz »

vladf wrote: post #30
thesz wrote: post #18

Earlier quoted context omitted.

I beg to disagree. [1] provides one with a whole-data-set training method (ADMM, one of such methods). Page 8 contains figure 2(b) - accuracy of training after specified amount of time. Note that ADMM start where stochastic gradient stops. [1] https://arxiv.org/pdf/1605.02026.pdf At [2] I tried to apply logistic regression trained using reweighted least squares algorithm on the same Higgs boson data set. I've got the…

> they do not quite care about parameter efficiency. Google Research is pretty big, I used to think like you did but I think it's mostly b/c DeepMind just hogs all the spotlight. Check out PRESS [0] for example. [0]: https://research.google/pubs/pub46141

Thank you! I skimmed over the abstract and will read the paper later, it seems interesting.

But you gave me another point to support my view: PRESS uses stochastic gradient, not second-order method like IRLS.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#59

Post by exikyut »

jsnell wrote: post #37
exikyut wrote: post #35

Earlier quoted context omitted.

Use x1e.32xlarge on AWS with 3TB of RAM. Just $12,742/mo - https://calculator.aws/#/estimate?id=7428fa81192c57087ac8cdf... Alternatively order something like the HP Z8 with 3TB RAM configured, which is only $75k - https://zworkstations.com/configurations/2040422/ It's interesting. It would take ~six years for the Z8 to break even compared to AWS, but traffic into and out of the machine would be $0, and I don't think…

> It would take ~six years for the Z8 to break even Do you mean six months?

Oh *dear*. I definitely tripped over there, and I didn't even notice.

Yup.

Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer

#60

Post by exikyut »

visarga wrote: post #38
exikyut wrote: post #35

Earlier quoted context omitted.

Use x1e.32xlarge on AWS with 3TB of RAM. Just $12,742/mo - https://calculator.aws/#/estimate?id=7428fa81192c57087ac8cdf... Alternatively order something like the HP Z8 with 3TB RAM configured, which is only $75k - https://zworkstations.com/configurations/2040422/ It's interesting. It would take ~six years for the Z8 to break even compared to AWS, but traffic into and out of the machine would be $0, and I don't think…

hate to break the party but this model only loads a small part of itself in RAM when inferencing

That's a good thing. Less completely means more energy for interestingness, and less expense means more accessibility.
Post reply on HN