Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OPT: Open Pre-trained Transformer Language Models

arxiv.org

Post reply on HN
Print view View original

141–150 of 242 posts

Re: OPT: Open Pre-trained Transformer Language Models

#141

Post by Rebelgecko »

yosito wrote: post #62
thorum wrote: post #31

A quick summary of the Limitations section: - "OPT-175B does not work well with declarative instructions or point-blank interrogatives." - "OPT-175B also tends to be repetitive and can easily get stuck in a loop. While sampling can reduce the incidence rate of repetitive behavior (Holtzman et al., 2020), we anecdotally found it did not eliminate it entirely when only one generation is sampled." - "We also find OPT-17…

> OPT-175B has a high propensity to generate toxic language and reinforce harmful stereotypes So they trained it on Facebook comments?

I thought Pushshift was only reddit comments?

Re: OPT: Open Pre-trained Transformer Language Models

#142

Post by rhizome »

ZephyrBlu wrote: post #10
ALittleLight wrote: post #5

Earlier quoted context omitted.

I don't like "available on request". I just want to download it and see if I can get it to run and mess around with it a bit. Why do I have to request anything? And I'm not an academic or researcher, so will they accept my random request? I'm also curious to know what the minimum requirements are to get this to run in inference mode.

Couple of random ideas: - They are concerned about the usage of the largest model, so want to vet people - The 175B parameter model is so large that it doesn't play nice with GitHub or something along those lines

I'd like to point the "Twitter suspensions are censorship!" people at this selective-participation filter.

Re: OPT: Open Pre-trained Transformer Language Models

#143

Post by arcticfox »

throwawayohio wrote: post #123
ad_hominem wrote: post #96

Earlier quoted context omitted.

Not handy, and I'm not going to spend my evening digging. It may've also been one of the NGOs ideologically aligned with him that credited him for the data + assistance

If it's so egregious is it really that hard to find an example of the bias? Calling the integrity of a single person operation into question, but then backing out with no evidence and even saying it might not have even been them seems a bit irresponsible.

On the other hand, they warned you with their username...

Re: OPT: Open Pre-trained Transformer Language Models

#144

Post by sanxiyn »

causality0 wrote: post #18

Out of curiosity, what's the file size on that?

175B * 16 bits = 350GB, but it does compress a bit.

GPT-J-6B, which you can download at https://github.com/kingoflolz/mesh-transformer-jax, is 6B parameters but weighs 9GB. It does decompress to 12GB as expected. Assuming the same compression ratio, download size would be 263GB, not 350GB.

Re: OPT: Open Pre-trained Transformer Language Models

#145

Post by Gigachad »

tsol wrote: post #130
Gigachad wrote: post #109

Earlier quoted context omitted.

I'd think any natural language model would have the same biases we see from real humans.

Are there really no moderated forums that the data can be taken from? Even HN-based training data would be much more civil

A model trained on HN would spit out a 5 paragraph story about how minorities provide a negative ROI for cities. Or how the homeless need to removed from society.

Re: OPT: Open Pre-trained Transformer Language Models

#146

Post by mike_d »

ad_hominem wrote: post #56
thorum wrote: post #31

A quick summary of the Limitations section: - "OPT-175B does not work well with declarative instructions or point-blank interrogatives." - "OPT-175B also tends to be repetitive and can easily get stuck in a loop. While sampling can reduce the incidence rate of repetitive behavior (Holtzman et al., 2020), we anecdotally found it did not eliminate it entirely when only one generation is sampled." - "We also find OPT-17…

> Pushshift.io Reddit corpus Pushshift is a single person with some very strong political opinions who has specifically used his datasets to attack political opponents. Frankly I wouldn't trust his data to be untainted. These models really need to be trained on more official data sources, or at least something with some type of multi-party oversight rather than data that effectively fell off the back of a truck. edit…

> Just look at the disastrous mess of his half-baked "opt-out" thing that flagrantly violates GDPR

Pushshift collects data from Reddit using the same API as the mobile app and public site. It does not have any privileged access to the Reddit database, nor is it collecting any PII that would be subject to GDPR.

You as a user grant a pretty broad license to Reddit when you post content. One of the things the license allows them to do is redistribute the content to other users as well as search indexes and things like the Wayback Machine or Pushshift.

(While I did work for Reddit at one point, these opinions are my own)

Re: OPT: Open Pre-trained Transformer Language Models

#147

Post by ALittleLight »

ipaddr wrote: post #133
vladf wrote: post #127

Earlier quoted context omitted.

Slightly modify a million random weights by changing the least significant bit up or down.

Compare three copies.

Or slightly randomly modify all the parameters on the copy you distribute, then it will be a match for nobody.

Re: OPT: Open Pre-trained Transformer Language Models

#148

Post by teaearlgraycold »

lumost wrote: post #37

I often wonder if OpenAIs decision not to open gpt-3 was because it was to expensive to train relative to its real value. They’ve hidden the model behind an api where they can filter out most of the dumb behaviors, while everyone believes they are working on something entirely different.

> They’ve hidden the model behind an api where they can filter out most of the dumb behaviors

What do you mean by this?

Re: OPT: Open Pre-trained Transformer Language Models

#149

Post by dotnet00 »

daenz wrote: post #45
p1esk wrote: post #43

Earlier quoted context omitted.

How can you identify content generated with them?

By training a GAN. A trained GAN will be able to accurately guess whether a block of text was produced by this GPT model, some other GPT model, or is authentic.

If differentiating between real samples and generated ones were as straightforward as "training a GAN", detecting deep fakes would not be as big of a research topic as it is.
Post reply on HN