Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OPT: Open Pre-trained Transformer Language Models

arxiv.org

Post reply on HN
Print view View original

131–140 of 242 posts

Re: OPT: Open Pre-trained Transformer Language Models

#131

Post by Vetch »

mikolajw wrote: post #42

The big one, OPT-175B, isn't an open model. The word "open" in technology means that everyone has equal access (viz. "open source software" and "open source hardware"). The article says that research access will be provided upon request for "academic researchers; those affiliated with organizations in government, civil society, and academia; and those in industry research laboratories.". Don't assume any good intent…

The other ones are smaller but not much worse according to their tests (oddly, in the Winograd Schema Challenge and Commitment Bank tasks, the largest model actually appears to be worse than much smaller ones).

30B parameter models are already large enough to exhibit some of the more interesting emergent phenomena of LLMs. Quantized to 8 bits, it might be possible to squeeze into 2, better three 3090s. But the models also seem undercooked, slightly to strongly under-performing GPT-3 in a lot of tasks. To further train the same model is now looking at > 100 GB, possibly 200GB of VRAM. Point being, this is no small thing they're offering and certainly preferable to being put on a waiting list for a paid API. The 6.7B and 13B parameter models seem the best bang for your buck as an individual.

Re: OPT: Open Pre-trained Transformer Language Models

#132

Post by alar44 »

ALittleLight wrote: post #121
alar44 wrote: post #100

Earlier quoted context omitted.

Gimme gimme. I want all your research and man hours for free. Gimme gimme. They are a for profit company and don't need to release anything. It's not that hard to understand.

Sure, but I'm an individual and free to say what I do and don't like. Why is that hard to understand?

Because it's a dumb thing to say. "Not really a fan of having to pay for my dinner!" It's just silly.

Re: OPT: Open Pre-trained Transformer Language Models

#134

Post by coffeeblack »

Jensson wrote: post #107
cortesoft wrote: post #103

Earlier quoted context omitted.

Well, the decision not to release the model might have been made so that they could license it instead.

They gave a reason why they didn't release it to the public, they said it was too dangerous: https://www.theguardian.com/technology/2019/feb/14/elon-musk... But of course then they started selling it to the highest bidder, so I wouldn't really trust what they say. They aren't "OpenAI", at this point they are just regular "ProprietaryAI". I really wonder what goal Elon Musk have with it.

Didn’t Musk leave the organization because they started doing things he didn’t like?

Re: OPT: Open Pre-trained Transformer Language Models

#135

Post by rhizome »

MengerSponge wrote: post #94
yosito wrote: post #62

Earlier quoted context omitted.

> OPT-175B has a high propensity to generate toxic language and reinforce harmful stereotypes So they trained it on Facebook comments?

Does it merely reinforce harmful stereotypes? Or will it help perpetrate genocide?

Tomato, tomahto.

Re: OPT: Open Pre-trained Transformer Language Models

#136

Post by dotnet00 »

londons_explore wrote: post #80
bogwog wrote: post #77

Earlier quoted context omitted.

That is dumb when you consider that this thing is likely going to leak anyways. It’s inevitable, and when it does happen, it will just end up in the hands of criminals/scammers and not the general public.

It's super easy to watermark weights for ML models. Just add a random 0.01 to a random weight anywhere in the network. It will have very little impact on the results, but will mean you can identify who leaked the weights.

It should be easy enough to make that sort of signature very difficult to trace by simply adding a bunch of small noise to the network overall, or even simply training for a few iterations.

Re: OPT: Open Pre-trained Transformer Language Models

#137

Post by sanxiyn »

tsol wrote: post #130
Gigachad wrote: post #109

Earlier quoted context omitted.

I'd think any natural language model would have the same biases we see from real humans.

Are there really no moderated forums that the data can be taken from? Even HN-based training data would be much more civil

Note that HN is included in the training data, see page 20.

Re: OPT: Open Pre-trained Transformer Language Models

#138

Post by SteveDR »

sigmoid10 wrote: post #128
d--b wrote: post #114

Remember when OpenAi wrote this? > Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT-2 along with sampling code. We are not releasing the dataset, training code, or GPT-2 model weights Well I guess Meta doesn’t care. https://openai.com/blog/better-language-models/

Ever since OpenAI transitioned away from the non-profit model, I'd take these statements with a grain if salt. Yes, there may also be some truth in that opinion, but don't underestimate monetary interests when someone has an easy ~12 month industry lead. Meta's existence and financial wellbeing on the other hand doesn't depend on this stuff, so they have less incentive to keep things proprietary. It seems ironic and…

I feel the same way. It does seem odd, though, that Meta would release this despite the precedent set by OpenAI with statements like this. What does Meta gain by releasing this for download?

Re: OPT: Open Pre-trained Transformer Language Models

#139

Post by wdroz »

d--b wrote: post #114

Remember when OpenAi wrote this? > Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT-2 along with sampling code. We are not releasing the dataset, training code, or GPT-2 model weights Well I guess Meta doesn’t care. https://openai.com/blog/better-language-models/

I hate the nanny point of view of OpenAI. IMO trashing Meta because theirs models may be misused isn't fair.

I think that hackers should advocate to have the freedom to toy/work with these models.

Re: OPT: Open Pre-trained Transformer Language Models

#140

Post by throwmeariver1 »

throwawayohio wrote: post #123
ad_hominem wrote: post #96

Earlier quoted context omitted.

Not handy, and I'm not going to spend my evening digging. It may've also been one of the NGOs ideologically aligned with him that credited him for the data + assistance

If it's so egregious is it really that hard to find an example of the bias? Calling the integrity of a single person operation into question, but then backing out with no evidence and even saying it might not have even been them seems a bit irresponsible.

You can just look at the data…
Post reply on HN