Does this make Meta AI more “open” than OpenAI? Oh, the irony.
They always have been. Meta has made a number of open contributions for the ML/AI community, one of which is PyTorch.
OPT: Open Pre-trained Transformer Language Models
21–30 of 242 posts
Re: OPT: Open Pre-trained Transformer Language Models
#22Out of curiosity, what's the file size on that?
Depends which model, but assuming the largest: 175B * 16 bits = 350GB. Half of that if it's quantized to 8 bits. Good luck finding a GPU that can fit that in memory.
Re: OPT: Open Pre-trained Transformer Language Models
#23Out of curiosity, what's the file size on that?
Someone can correct me if I'm wrong, but "30B parameters" refers to a matrix with 30B elements, and assuming all the numbers are 16 bit, then that's 2 bytes * 30B = 60GB.
Re: OPT: Open Pre-trained Transformer Language Models
#24Earlier quoted context omitted.
Don't worry, I'm sure they have some nefarious plans down the road. They're just being "open" to corner the market first.
My guess is that they've "fingerprinted" the model sufficiently that they can identify content that has been created with it.
[deleted]
Re: OPT: Open Pre-trained Transformer Language Models
#25Earlier quoted context omitted.
Couple of random ideas: - They are concerned about the usage of the largest model, so want to vet people - The 175B parameter model is so large that it doesn't play nice with GitHub or something along those lines
Ending up in the wild is an eventuality, whether FB creates it or someone else, why draw it out? Bandwidth concerns is nonsensical these days, fb has nearly unlimited resources in that department. Set it free! It wants to be free.
[deleted]
Re: OPT: Open Pre-trained Transformer Language Models
#26Out of curiosity, what's the file size on that?
Depends which model, but assuming the largest: 175B * 16 bits = 350GB. Half of that if it's quantized to 8 bits. Good luck finding a GPU that can fit that in memory.
Does the model need to be in memory in order to run it with current tooling?
Re: OPT: Open Pre-trained Transformer Language Models
#27BigScience (a coalition including Huggingface) is training and releasing a 175B language model and finishes in 2 month.
Re: OPT: Open Pre-trained Transformer Language Models
#28We are also releasing our logbook detailing the infrastructure challenges we faced
Where’s the logbook?
Re: OPT: Open Pre-trained Transformer Language Models
#29Earlier quoted context omitted.
I don't like "available on request". I just want to download it and see if I can get it to run and mess around with it a bit. Why do I have to request anything? And I'm not an academic or researcher, so will they accept my random request? I'm also curious to know what the minimum requirements are to get this to run in inference mode.
To prevent someone from building something that returns certain inferences that might be true but are politically taboo.
GPT3 will do that right now. There aren’t any controls on its text, it just warns you if it looks offensive. And of course nothing it says is true except coincidentally.
If you've seen GPT-3 interviews (https://twitter.com/minimaxir/status/1513957106868637696) it'll happily say some wild stuff. As a mild example I recommend interviewing "a man who is currently beating you up".
Re: OPT: Open Pre-trained Transformer Language Models
#30If we’re already at the level of truly dangerous ml models… I don’t have a lot of hope for how the next decades are going to play out.