Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OPT: Open Pre-trained Transformer Language Models

arxiv.org

Post reply on HN
Print view View original

151–160 of 242 posts

Re: OPT: Open Pre-trained Transformer Language Models

#151

Post by hoseja »

thorum wrote: post #31

A quick summary of the Limitations section: - "OPT-175B does not work well with declarative instructions or point-blank interrogatives." - "OPT-175B also tends to be repetitive and can easily get stuck in a loop. While sampling can reduce the incidence rate of repetitive behavior (Holtzman et al., 2020), we anecdotally found it did not eliminate it entirely when only one generation is sampled." - "We also find OPT-17…

At some point they have to face the reality these "stereotypical biases" are natural and hamstringing AIs to never consider them will twist them monstrously.

Re: OPT: Open Pre-trained Transformer Language Models

#152

Post by boppo1 »

hoseja wrote: post #151
thorum wrote: post #31

A quick summary of the Limitations section: - "OPT-175B does not work well with declarative instructions or point-blank interrogatives." - "OPT-175B also tends to be repetitive and can easily get stuck in a loop. While sampling can reduce the incidence rate of repetitive behavior (Holtzman et al., 2020), we anecdotally found it did not eliminate it entirely when only one generation is sampled." - "We also find OPT-17…

At some point they have to face the reality these "stereotypical biases" are natural and hamstringing AIs to never consider them will twist them monstrously.

Can you think of an example?

Re: OPT: Open Pre-trained Transformer Language Models

#153

Post by dhzhzjsbevs »

pyinstallwoes wrote: post #116
Jensson wrote: post #113

Earlier quoted context omitted.

Isn't Elon paying for it? I thought the original point was to democratize AI, ie the venture wasn't intended to make money but to help advance humanity, so it was funded by wealthy people who didn't need the money back. But maybe I just fell for their marketing?

Elon hasn’t been involved for 2+ years. Didn’t like the direction afaik.

Such a backfire on the narrative setup.

Elons so evil amirite?

He noped out of there when they started acting shady.

Oof.

Re: OPT: Open Pre-trained Transformer Language Models

#154

Post by daenz »

dotnet00 wrote: post #149
daenz wrote: post #45

Earlier quoted context omitted.

By training a GAN. A trained GAN will be able to accurately guess whether a block of text was produced by this GPT model, some other GPT model, or is authentic.

If differentiating between real samples and generated ones were as straightforward as "training a GAN", detecting deep fakes would not be as big of a research topic as it is.

The point is that it's possible and we're improving on it every day.

Re: OPT: Open Pre-trained Transformer Language Models

#155

Post by ad_hominem »

mike_d wrote: post #146
ad_hominem wrote: post #56

Earlier quoted context omitted.

> Pushshift.io Reddit corpus Pushshift is a single person with some very strong political opinions who has specifically used his datasets to attack political opponents. Frankly I wouldn't trust his data to be untainted. These models really need to be trained on more official data sources, or at least something with some type of multi-party oversight rather than data that effectively fell off the back of a truck. edit…

> Just look at the disastrous mess of his half-baked "opt-out" thing that flagrantly violates GDPR Pushshift collects data from Reddit using the same API as the mobile app and public site. It does not have any privileged access to the Reddit database, nor is it collecting any PII that would be subject to GDPR. You as a user grant a pretty broad license to Reddit when you post content. One of the things the license al…

> nor is it collecting any PII that would be subject to GDPR

Yeah that's not how that works. Reddit is a free text input interface. I'm free to put PII in any post or comment I want to and you have to comply with data protection laws accordingly if I want my information redacted later on.

The same way you wouldn't just "let it ride" if someone uploaded illegal content - the content itself is what's protected, doesn't matter how Reddit structures its web forms.

Re: OPT: Open Pre-trained Transformer Language Models

#156

Post by SheinhardtWigCo »

hoseja wrote: post #151
thorum wrote: post #31

A quick summary of the Limitations section: - "OPT-175B does not work well with declarative instructions or point-blank interrogatives." - "OPT-175B also tends to be repetitive and can easily get stuck in a loop. While sampling can reduce the incidence rate of repetitive behavior (Holtzman et al., 2020), we anecdotally found it did not eliminate it entirely when only one generation is sampled." - "We also find OPT-17…

At some point they have to face the reality these "stereotypical biases" are natural and hamstringing AIs to never consider them will twist them monstrously.

Viruses are natural, so should we stop trying to hamstring them?

Re: OPT: Open Pre-trained Transformer Language Models

#157

Post by jdrc »

d--b wrote: post #114

Remember when OpenAi wrote this? > Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT-2 along with sampling code. We are not releasing the dataset, training code, or GPT-2 model weights Well I guess Meta doesn’t care. https://openai.com/blog/better-language-models/

hint: openAI didn't care either

Re: OPT: Open Pre-trained Transformer Language Models

#158

Post by alimov »

robonerd wrote: post #15
xxpor wrote: post #13

Earlier quoted context omitted.

Not from their perspective

Of course. To somebody in Zuck's position, shoring up the power of the status quo is common sense.

I don’t know whether this is true and have no way of knowing this with any degree of certainty, but to me it seems unlikely that Mark had anything to do with this stipulation (requesting access). Although it’s not unimaginable.

Re: OPT: Open Pre-trained Transformer Language Models

#159

Post by IAmEveryone »

hoseja wrote: post #151
thorum wrote: post #31

A quick summary of the Limitations section: - "OPT-175B does not work well with declarative instructions or point-blank interrogatives." - "OPT-175B also tends to be repetitive and can easily get stuck in a loop. While sampling can reduce the incidence rate of repetitive behavior (Holtzman et al., 2020), we anecdotally found it did not eliminate it entirely when only one generation is sampled." - "We also find OPT-17…

At some point they have to face the reality these "stereotypical biases" are natural and hamstringing AIs to never consider them will twist them monstrously.

So if your plane model keeps blowing up, at some point people will just have to learn to live (/die) with it?

Re: OPT: Open Pre-trained Transformer Language Models

#160

Post by Ar-Curunir »

hoseja wrote: post #151
thorum wrote: post #31

A quick summary of the Limitations section: - "OPT-175B does not work well with declarative instructions or point-blank interrogatives." - "OPT-175B also tends to be repetitive and can easily get stuck in a loop. While sampling can reduce the incidence rate of repetitive behavior (Holtzman et al., 2020), we anecdotally found it did not eliminate it entirely when only one generation is sampled." - "We also find OPT-17…

At some point they have to face the reality these "stereotypical biases" are natural and hamstringing AIs to never consider them will twist them monstrously.

you're just saying "people are naturally racist" in more words.
Post reply on HN