Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer
1–10 of 78 posts
Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer
#2lol they didn’t open source the model weights
Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer
#3lol they didn’t open source the model weights
hah! yeah that's what I was looking for too
Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer
#4lol they didn’t open source the model weights
data is the new oil, what's the analogy for the data industry's impact on society akin climate change?
Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer
#5lol they didn’t open source the model weights
data is the new oil, what's the analogy for the data industry's impact on society akin climate change?
Surveillance Capitalism
Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer
#6lol they didn’t open source the model weights
In the source code it says "I have discovered truly marvelous weights for this, which this header file is too small to contain"
Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer
#7If the model weights aren’t open source, the model isn’t open source in my opinion.
It’s not like a switch transformer is that much more complicated to slap into pytorch than any other new research.
Also in my personal opinion, while yes this model performs great, it’s ridiculously parameter inefficient. 1.6T parameters is 3.12TB using bfloat 16. I’d rather take 7x longer to train and have a model I can actually put places.
Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer
#8lol they didn’t open source the model weights
If I did the math right it would be 3.12TB of weights, maybe they are trying to upload it to gdrive still. (/s, probably)
Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer
#9If the model weights aren’t open source, the model isn’t open source in my opinion. It’s not like a switch transformer is that much more complicated to slap into pytorch than any other new research. Also in my personal opinion, while yes this model performs great, it’s ridiculously parameter inefficient. 1.6T parameters is 3.12TB using bfloat 16. I’d rather take 7x longer to train and have a model I can actually put…
> it’s ridiculously parameter inefficient
How do you know that? Perhaps with this methodology you really need those 3.12TB to reach comparable performance?
Re: Google Open-Sources Trillion-Parameter AI Language Model Switch Transformer
#10lol they didn’t open source the model weights
If I did the math right it would be 3.12TB of weights, maybe they are trying to upload it to gdrive still. (/s, probably)
They released more data than that for their Google Books n-grams datasets:
https://storage.googleapis.com/books/ngrams/books/datasetsv3...
(I don't remember exactly how much it is, but I remember that the old version was already in the terabytes.)