31–40 of 78 posts
Would GDPR or other regulation apply to the data that these model is trained with? Is it not a risk that the model will record some private information in someone's email?
Would GDPR or other regulation apply to the data that these model is trained with? Is it not a risk that the model will record some private information in someone's email?
Yes, it has been shown [1] that these models can memorize information even if it only appears once in the training data. It could potentially cause issues with privacy and especially copyright. However, in this case it's not relevant because they didn't release the model weights. [1] https://arxiv.org/abs/2012.07805
lol they didn’t open source the model weights
Is this because they are afraid of the model misused, like used for generating fake reviews? It is frustrating that I've been hearing great news on NLP but am able to try none of them myself.
lol they didn’t open source the model weights
If I did the math right it would be 3.12TB of weights, maybe they are trying to upload it to gdrive still. (/s, probably)
This model has 3.12TB of floats??? That's insane. How do you load that into memory for inferencing?
Earlier quoted context omitted.
If I did the math right it would be 3.12TB of weights, maybe they are trying to upload it to gdrive still. (/s, probably)
Weights are just numbers (probably floats?), right? This model has 3.12TB of floats??? That's insane. How do you load that into memory for inferencing?
Alternatively order something like the HP Z8 with 3TB RAM configured, which is only $75k - https://zworkstations.com/configurations/2040422/
It's interesting. It would take ~six years for the Z8 to break even compared to AWS, but traffic into and out of the machine would be $0, and I don't think you're running directly on the metal with AWS, so performance would probably be a bit higher. And then there's storage - I configured, uhh, 120TB of a mixture of SSDs and HDDs. I'm not even going to try and ask AWS for a comparible quote there.
I may or may not have added dual Xeon Platinum 8280s to the Z8 as well. :P
The brain "works" because it's evolved structure matches or reflects reality. It is not about having billions of neurons, but about to have the right structure which matches the environment.
My favourite example is how butterflies evolve pictures of eyes on its wings to scare predators, having literally no idea about existence of other creatures.
It has been evolved because other creatures have eyes, and they are there, of course.
The proper structure of neural networks must be based on such fundamental features, like "most of creatures have eyes" and similar ones.
Brain does not have a flat structure, like a billion x billion matrix. It is more clever and simpler that that.
A language model must be based on the fundamental notion that there are nouns (things), verbs (processes) and adjectives (attributes). It is that simple.
Earlier quoted context omitted.
Weights are just numbers (probably floats?), right? This model has 3.12TB of floats??? That's insane. How do you load that into memory for inferencing?
Use x1e.32xlarge on AWS with 3TB of RAM. Just $12,742/mo - https://calculator.aws/#/estimate?id=7428fa81192c57087ac8cdf... Alternatively order something like the HP Z8 with 3TB RAM configured, which is only $75k - https://zworkstations.com/configurations/2040422/ It's interesting. It would take ~six years for the Z8 to break even compared to AWS, but traffic into and out of the machine would be $0, and I don't think…
Do you mean six months?
Earlier quoted context omitted.
Weights are just numbers (probably floats?), right? This model has 3.12TB of floats??? That's insane. How do you load that into memory for inferencing?
Use x1e.32xlarge on AWS with 3TB of RAM. Just $12,742/mo - https://calculator.aws/#/estimate?id=7428fa81192c57087ac8cdf... Alternatively order something like the HP Z8 with 3TB RAM configured, which is only $75k - https://zworkstations.com/configurations/2040422/ It's interesting. It would take ~six years for the Z8 to break even compared to AWS, but traffic into and out of the machine would be $0, and I don't think…
So, almost every possible ngram ever used is a 'input'? Can someone describe what the lexical reality of 3.5T inputs actually means? I feel like this is 'Deep Memorization' instead of 'Deep Learning'. Like a Doctor who passes everything merely by memorizing the textbook with absolutely no ability beyond that.
Earlier quoted context omitted.
Yes, it has been shown [1] that these models can memorize information even if it only appears once in the training data. It could potentially cause issues with privacy and especially copyright. However, in this case it's not relevant because they didn't release the model weights. [1] https://arxiv.org/abs/2012.07805
Even if they did not, they have it for themselves. If this uses any personal data, then GDPR and other regulations may not be getting respected.