Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OPT: Open Pre-trained Transformer Language Models

arxiv.org

Post reply on HN
Print view View original

51–60 of 242 posts

Re: OPT: Open Pre-trained Transformer Language Models

#51

Post by daenz »

voz_ wrote: post #50
daenz wrote: post #38

Earlier quoted context omitted.

It's pretty simple. GPT models are essentially information weapons. People are going to get their hands on them, so might as well give them a model where you can identify content generated with them, so you can know who is using them for nefarious purposes. Like how many printers encode hidden patterns on paper that identify the model of the printer and other information[0] 0. https://www.bbc.com/future/article/20170…

This is nonsense.

Would an AI @ FB employee admit it if it was true?

Re: OPT: Open Pre-trained Transformer Language Models

#53

Post by ineedasername »

dukeofdoom wrote: post #33
bestcoder69 wrote: post #19

Earlier quoted context omitted.

You think GPT-3 generates text that's truthful? Have you used it even once?

I haven't used GPT-3, but I did try out a site that was based on GPT2. I believe it was called "talk to transformer". But I never tried quarrying anything controversial. However, I bet this a concern and certain queries will be filtered or "corrected" to be more politically correct. To give you an example, a few days ago I made a comment one Alex Jones, and wanted to google him. The second link returned on him was fr…

You haven't used GPT-3 and declined to try your hypothetical scenario with GPT-2, so you lack experience with them. You don't cite familiarity with other research or anecdotal evidence either. So what exactly is your justification here? Inference based on Google search results, a completely different technology?

Re: OPT: Open Pre-trained Transformer Language Models

#54

Post by speed_spread »

thorum wrote: post #31

A quick summary of the Limitations section: - "OPT-175B does not work well with declarative instructions or point-blank interrogatives." - "OPT-175B also tends to be repetitive and can easily get stuck in a loop. While sampling can reduce the incidence rate of repetitive behavior (Holtzman et al., 2020), we anecdotally found it did not eliminate it entirely when only one generation is sampled." - "We also find OPT-17…

Reminds me a lot of "Do not taunt Happy Fun Ball".

Re: OPT: Open Pre-trained Transformer Language Models

#55

Post by voz_ »

daenz wrote: post #51
voz_ wrote: post #50

Earlier quoted context omitted.

This is nonsense.

Would an AI @ FB employee admit it if it was true?

> I will never discuss FB technical details, internals, or anything else on this site, so please do not ask.

My claim of nonsense has nothing to do with FB. You cannot fingerprint models like this, that's just not how it works.

Also, if we are reading profiles, you call yourself a 10x engineer on your blog, that's hilarious. Maybe 10x the nonsense?

Re: OPT: Open Pre-trained Transformer Language Models

#56

Post by ad_hominem »

thorum wrote: post #31

A quick summary of the Limitations section: - "OPT-175B does not work well with declarative instructions or point-blank interrogatives." - "OPT-175B also tends to be repetitive and can easily get stuck in a loop. While sampling can reduce the incidence rate of repetitive behavior (Holtzman et al., 2020), we anecdotally found it did not eliminate it entirely when only one generation is sampled." - "We also find OPT-17…

> Pushshift.io Reddit corpus

Pushshift is a single person with some very strong political opinions who has specifically used his datasets to attack political opponents. Frankly I wouldn't trust his data to be untainted.

These models really need to be trained on more official data sources, or at least something with some type of multi-party oversight rather than data that effectively fell off the back of a truck.

edit: That's not even to mention I believe it's flat-out illegal for him to collect and redistribute this data as Reddit users did not agree to any terms of use with him. Just look at the disastrous mess of his half-baked "opt-out" thing that flagrantly violates GDPR: https://www.reddit.com/r/pushshift/comments/pat409/online_re...

Re: OPT: Open Pre-trained Transformer Language Models

#57

Post by daenz »

voz_ wrote: post #55
daenz wrote: post #51

Earlier quoted context omitted.

Would an AI @ FB employee admit it if it was true?

> I will never discuss FB technical details, internals, or anything else on this site, so please do not ask. My claim of nonsense has nothing to do with FB. You cannot fingerprint models like this, that's just not how it works. Also, if we are reading profiles, you call yourself a 10x engineer on your blog, that's hilarious. Maybe 10x the nonsense?

>You cannot fingerprint models like this

A GAN can absolutely be trained to discriminate between text generated from this model or another model.

>that's hilarious

What's hilarious about it?

Re: OPT: Open Pre-trained Transformer Language Models

#59

Post by sanxiyn »

metadat wrote: post #17
ZephyrBlu wrote: post #10

Earlier quoted context omitted.

Couple of random ideas: - They are concerned about the usage of the largest model, so want to vet people - The 175B parameter model is so large that it doesn't play nice with GitHub or something along those lines

Ending up in the wild is an eventuality, whether FB creates it or someone else, why draw it out? Bandwidth concerns is nonsensical these days, fb has nearly unlimited resources in that department. Set it free! It wants to be free.

"It wants to be free" is a ridiculous statement, considering that after full two years (GPT-3 was published in May 2020), there is no public release of anything comparable.

In May 2020, was your estimate of time to public release of anything comparable shorter or longer than two years? I bet it was shorter.

Re: OPT: Open Pre-trained Transformer Language Models

#60

Post by voz_ »

daenz wrote: post #57
voz_ wrote: post #55

Earlier quoted context omitted.

> I will never discuss FB technical details, internals, or anything else on this site, so please do not ask. My claim of nonsense has nothing to do with FB. You cannot fingerprint models like this, that's just not how it works. Also, if we are reading profiles, you call yourself a 10x engineer on your blog, that's hilarious. Maybe 10x the nonsense?

>You cannot fingerprint models like this A GAN can absolutely be trained to discriminate between text generated from this model or another model. >that's hilarious What's hilarious about it?

> A GAN can absolutely be trained to discriminate between text generated from this model or another model.

Nope. I dare you to do it. Or at least intelligently articulate the model architectures for doing so.

> What's hilarious about it?

It's a bullshit term, firstoff, and calling yourself that is the height of ego. Might as well throw in rockstar, ninja, etc too.

Post reply on HN