Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OPT: Open Pre-trained Transformer Language Models

arxiv.org

Post reply on HN
Print view View original

201–210 of 242 posts

Re: OPT: Open Pre-trained Transformer Language Models

#201

Post by bestcoder69 »

dukeofdoom wrote: post #33
bestcoder69 wrote: post #19

Earlier quoted context omitted.

You think GPT-3 generates text that's truthful? Have you used it even once?

I haven't used GPT-3, but I did try out a site that was based on GPT2. I believe it was called "talk to transformer". But I never tried quarrying anything controversial. However, I bet this a concern and certain queries will be filtered or "corrected" to be more politically correct. To give you an example, a few days ago I made a comment one Alex Jones, and wanted to google him. The second link returned on him was fr…

OK I'll answer you, but I want you to introspect on your bet. What if you're 100% wrong? What would it mean about your priors? Think about that before continuing, if you're capable. Really stop and think about this...

...

...

...

Alright welcome back. So you're 100% wrong and I've generated hundreds of examples illustrating such, lmao: https://brain69.substack.com/

Re: OPT: Open Pre-trained Transformer Language Models

#202

Post by bestcoder69 »

thorum wrote: post #31

A quick summary of the Limitations section: - "OPT-175B does not work well with declarative instructions or point-blank interrogatives." - "OPT-175B also tends to be repetitive and can easily get stuck in a loop. While sampling can reduce the incidence rate of repetitive behavior (Holtzman et al., 2020), we anecdotally found it did not eliminate it entirely when only one generation is sampled." - "We also find OPT-17…

> - "OPT-175B does not work well with declarative instructions or point-blank interrogatives."

Lame!!! I've come to realize InstructGPT3 is just so so so much better than base GPT-3. I won't be _too_ excited about competitors yet until someone makes their own instruct model.

Re: OPT: Open Pre-trained Transformer Language Models

#203

Post by bestcoder69 »

LeicaLatte wrote: post #36

As someone who finds openai patronizing, this is welcome.

I love text-davinci-002, but they need competition, badly. Their ToS is preventing me from releasing the world's greatest chatbot :P https://old.reddit.com/r/GPT3/comments/ubm0hm/my_customizabl...

Re: OPT: Open Pre-trained Transformer Language Models

#205

Post by saynay »

ALittleLight wrote: post #5
MasterScrat wrote: post #2

"We are releasing all of our models between 125M and 30B parameters, and will provide full research access to OPT-175B upon request. Access will be granted to academic researchers; those affiliated with organizations in government, civil society, and academia; and those in industry research laboratories." GPT-3 Davinci ("the" GPT-3) is 175B. The repository will be open "First thing in AM" ( https://twitter.com/stephe…

I don't like "available on request". I just want to download it and see if I can get it to run and mess around with it a bit. Why do I have to request anything? And I'm not an academic or researcher, so will they accept my random request? I'm also curious to know what the minimum requirements are to get this to run in inference mode.

If it is like many other models, part of the reason would just be to reduce their bandwidth costs. The models can be huge, and they want to limit those who just want to download it on a whim so they don't rack up $10k+ is bandwidth charges, as has happened to many others who hosted big models out on S3 or something.

Re: OPT: Open Pre-trained Transformer Language Models

#206

Post by sireat »

Vetch wrote: post #131
mikolajw wrote: post #42

The big one, OPT-175B, isn't an open model. The word "open" in technology means that everyone has equal access (viz. "open source software" and "open source hardware"). The article says that research access will be provided upon request for "academic researchers; those affiliated with organizations in government, civil society, and academia; and those in industry research laboratories.". Don't assume any good intent…

The other ones are smaller but not much worse according to their tests (oddly, in the Winograd Schema Challenge and Commitment Bank tasks, the largest model actually appears to be worse than much smaller ones). 30B parameter models are already large enough to exhibit some of the more interesting emergent phenomena of LLMs. Quantized to 8 bits, it might be possible to squeeze into 2, better three 3090s. But the models…

Can you actually stack multiple 3090s arbitrarily like that?

That is use multiple 3090s to load a single model for inference.

I thought that at most you could use two 3090s via NVlink.

Stacking multiple cards would open some real cheap options.

Like a real budget option would be something like a few ancient K80s (24GB version). eBay price was around $200-300 last I checked. .

Re: OPT: Open Pre-trained Transformer Language Models

#207

Post by [deleted] »

dukeofdoom wrote: post #33
bestcoder69 wrote: post #19

Earlier quoted context omitted.

You think GPT-3 generates text that's truthful? Have you used it even once?

I haven't used GPT-3, but I did try out a site that was based on GPT2. I believe it was called "talk to transformer". But I never tried quarrying anything controversial. However, I bet this a concern and certain queries will be filtered or "corrected" to be more politically correct. To give you an example, a few days ago I made a comment one Alex Jones, and wanted to google him. The second link returned on him was fr…

[deleted]

Re: OPT: Open Pre-trained Transformer Language Models

#208

Post by therealcamino »

dukeofdoom wrote: post #61
astrange wrote: post #29

Earlier quoted context omitted.

GPT3 will do that right now. There aren’t any controls on its text, it just warns you if it looks offensive. And of course nothing it says is true except coincidentally. If you've seen GPT-3 interviews ( https://twitter.com/minimaxir/status/1513957106868637696 ) it'll happily say some wild stuff. As a mild example I recommend interviewing "a man who is currently beating you up".

Is it true to say they are true coincidentally, because that kind of suggests randomly true. I understand the AI doesn't really comprehend if something is true or false. My understanding is the results are more than random, maybe something closer to like weighted opinion.

What it returns is based on what it's trained on. If it's trained on a corpus containing untruths and prejudice, you can get untruths and prejudice out. You can't make conclusions about what beliefs are widely held based on what it generates in response to specific prompts.

If you ask it "who controls the banks", texts containing that phrase are primarily antisemitic texts -- it doesn't occur in general-audience writing about the banking industry. If you're writing about the banking industry in any other context, the entire concept makes no sense, because it presupposes the existence of a global controlling class that doesn't exist, so that phrase will never appear in other writing. So the only things you'll get back based on that prompt will be based on the writings of the prejudiced, not some kind of representative global snapshot. Taking that as evidence of "weighted opinion" doesn't make sense.

Re: OPT: Open Pre-trained Transformer Language Models

#209

Post by JackC »

ALittleLight wrote: post #5
MasterScrat wrote: post #2

"We are releasing all of our models between 125M and 30B parameters, and will provide full research access to OPT-175B upon request. Access will be granted to academic researchers; those affiliated with organizations in government, civil society, and academia; and those in industry research laboratories." GPT-3 Davinci ("the" GPT-3) is 175B. The repository will be open "First thing in AM" ( https://twitter.com/stephe…

I don't like "available on request". I just want to download it and see if I can get it to run and mess around with it a bit. Why do I have to request anything? And I'm not an academic or researcher, so will they accept my random request? I'm also curious to know what the minimum requirements are to get this to run in inference mode.

> Why do I have to request anything? And I'm not an academic or researcher, so will they accept my random request?

Just a guess: you will have to contractually agree to some things in order to get the model; at a minimum, agree not to redistribute it, but probably also agree not to use it commercially. That means whatever commercial advantage there is to having a model this size isn't affected by this offer, which makes it lower stakes for Facebook to offer. And then the point of "academics and researchers" is to be a proxy for "people we trust to keep their promise because they have a clear usecase for non-commercial access to the model and a reputation to protect." They can also sue after the fact, but they'd rather not have to.

Not saying any of this is good or bad, just an educated guess about why it works the way it does.

Post reply on HN