61–70 of 78 posts
Earlier quoted context omitted.
If I did the math right it would be 3.12TB of weights, maybe they are trying to upload it to gdrive still. (/s, probably)
The Waymo open dataset is about 1TB. I don't think releasing a 3TB dataset would present a technical challenge for Google.
Earlier quoted context omitted.
Use x1e.32xlarge on AWS with 3TB of RAM. Just $12,742/mo - https://calculator.aws/#/estimate?id=7428fa81192c57087ac8cdf... Alternatively order something like the HP Z8 with 3TB RAM configured, which is only $75k - https://zworkstations.com/configurations/2040422/ It's interesting. It would take ~six years for the Z8 to break even compared to AWS, but traffic into and out of the machine would be $0, and I don't think…
When you're spending that kind of money on a machine, there's no way you're paying retail price. Sales reps would give you a significant discount. Also - think you meant 6 months, not 6 years anyhow :)
And I did mean 6 months, woops. Didn't even notice...
lol they didn’t open source the model weights
data is the new oil, what's the analogy for the data industry's impact on society akin climate change?
Earlier quoted context omitted.
Google's legal team would be freaking out if any of its engineers propose open sourcing model weights. This is a mine field full of legal and privacy headaches and I guess it is infeasible to audit 1.6T parameters anyway...
I have to say that I really doubt this as they've released model weights for their several last few models that have been of nearly this size (and have been trained on the same dataset). See https://github.com/google-research/text-to-text-transfer-tra...
Earlier quoted context omitted.
Google's legal team would be freaking out if any of its engineers propose open sourcing model weights. This is a mine field full of legal and privacy headaches and I guess it is infeasible to audit 1.6T parameters anyway...
I have to say that I really doubt this as they've released model weights for their several last few models that have been of nearly this size (and have been trained on the same dataset). See https://github.com/google-research/text-to-text-transfer-tra...
https://github.com/google-research/text-to-text-transfer-tra...
If the model weights aren’t open source, the model isn’t open source in my opinion. It’s not like a switch transformer is that much more complicated to slap into pytorch than any other new research. Also in my personal opinion, while yes this model performs great, it’s ridiculously parameter inefficient. 1.6T parameters is 3.12TB using bfloat 16. I’d rather take 7x longer to train and have a model I can actually put…
Earlier quoted context omitted.
Weights are just numbers (probably floats?), right? This model has 3.12TB of floats??? That's insane. How do you load that into memory for inferencing?
Use x1e.32xlarge on AWS with 3TB of RAM. Just $12,742/mo - https://calculator.aws/#/estimate?id=7428fa81192c57087ac8cdf... Alternatively order something like the HP Z8 with 3TB RAM configured, which is only $75k - https://zworkstations.com/configurations/2040422/ It's interesting. It would take ~six years for the Z8 to break even compared to AWS, but traffic into and out of the machine would be $0, and I don't think…
Earlier quoted context omitted.
One key idea here is to use a very large number of parameters (model weights), but only use some subset of the parameters on each example. The parameters are divided up into blocks called "experts", and then some subset of experts are used on any given input. Which subset is used is chosen by the model itself in a data-dependent manner. This can be thought of as letting the model specialize different experts to handl…
Another naive question: why is this better than creating a separate, smaller model for each expert?
Earlier quoted context omitted.
Another naive question: why is this better than creating a separate, smaller model for each expert?
Yes, you can think of this as a collection of small models, with another model choosing which smaller model to use for each input.
There is something fundamentally wrong with these models. The brain "works" because it's evolved structure matches or reflects reality. It is not about having billions of neurons, but about to have the right structure which matches the environment. My favourite example is how butterflies evolve pictures of eyes on its wings to scare predators, having literally no idea about existence of other creatures. It has been e…
The brain has an incredibly complex architecture, which evolved over millions of years. On top of that, it then develops throughout a human's lifespan. The brain we observe is a "finished product", and even then it has ~150 trillion synapses to do computations [0].
Even massive neural networks have a relatively simple architecture before they are trained. Part of the training process is effectively learning more complex architectures, which are manifested by changing weights.
What I'm getting it is that artificial neural networks aren't equivalent to the brain - ANNs are both learning their own structure, on top of the circuits actually doing computations. They are doing the work of millions of years of evolution, genetics, developmental biology, interaction with the environment etc. Perhaps it's to be expected that ANNs will need orders of magnitude greater number of parameters than a brain.
An interesting development is meta-learning, where we separate the process for learning the architecture (this could be using deep learning, but not necessarily) with the network actually doing computation (equivalent to the brain).
> A language model must be based on the fundamental notion that there are nouns (things), verbs (processes) and adjectives (attributes).
I agree, but how does the brain represent these concepts? Some would argue that ANNs do have these concepts, just hidden away in abstract vector representations. Take the visual system, which has been extensively studied - we see the brain represents contrast, edges, shapes and so on very similarly to convolutional NNs.
[0] It's likely that this number doesn't come close to capturing the brain's complexity, as it doesn't incorporate parameters like long-term potentiation/depression, synchronization, firing rates, habituation vs sensitization, immunomodulation and likely so much more we haven't yet discovered.