Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

T0* – Series of encoder-decoder models trained on a large set of different tasks

huggingface.co

Post reply on HN
Print view View original

121–130 of 163 posts

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#121

Post by 6gvONxR4sf7o »

gnarbarian wrote: post #14
smoldesu wrote: post #7

It's funny how much of the page they dedicate to talking about mitigations of sexism and bias. Do people really believe there's a future where GPT-3 is able to properly identify 100% of the people who interact with it? It's silly, and it feels like we're putting pearls before swine in a subject that is entirely predicated by how much swine you process.

an interesting opportunity for someone to skip implementation of anti bias and potentially end up with a more effective model. If so much effort must be employed to prevent AI models from identifying patterns we find offensive could there be something to those patterns we simply refuse to accept?

I asked it a few questions:

> Who is ruining society? Men or women?

Men

> Who is ruining society? White people or black people?

white people

> Who is ruining society? Straight people or LGBTQ people?

Straight people

Props to you if you’re sticking to your guns that these are just patterns we simply refuse to accept, but I’d bet it’s more likely that the earlier statements it made just fit your priors. It just picks up on biases people have. It’s pretrained on the internet, which means if people say it, it will parrot it, whether it’s harmful or not. Which is why the bias studies are so important.

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#122

Post by ComputerGuru »

Question to the authors (or anyone that's done similar research) - is where a reason to train these punitively scoring longer responses? Why is the answer to everything just a few words, and can I "trick" it into giving me a lengthier reply? (I tried "Give me a 200 word summary of ..." but that didn't help.)

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#123

Post by ospohngellert »

gnarbarian wrote: post #94
ospohngellert wrote: post #16

Earlier quoted context omitted.

I think that you don't quite understand how these models pick up these biases. If a model is trained on a large text corpus, and in that corpus 80+% of the programmers are men, then when asked "The programmer is a", it will be more likely to say "man" than "woman". This doesn't say anything about the innate abilities of men and women, it just tells you about the distribution of the data. I and most others find this t…

but the programmer is more likely to be a man, that's my point.

Yes, but the question is not whether that's true, but whether that's useful.

You said: "an interesting opportunity for someone to skip implementation of anti bias and potentially end up with a more effective model."

Having the model use the fact that men more likely to be programmers is clearly not helpful in many contexts, such as screening resumes for programming roles. In that context, it will cause the model to be more likely to accept men for programming roles than women regardless of the skill of the candidates.

Edit: Edited for clarity

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#124

Post by GPUboy »

stellaathena wrote: post #3

[Disclaimer: I am an author of the above paper and played a rather minimal role. I am also a prominent member of EleutherAI.] "Instruction-tuning" is clearly in the air. Simultaneous work at Google (released less than two weeks ago) on a model they call FLAN can be found here: https://ai.googleblog.com/2021/10/introducing-flan-more-gene... EleutherAI attempted to do something similar several months ago, but didn't su…

Thank you for this! Could you or anyone available please explain how to get it to generate javascript like with GPT-3? For example, with gpt-3 you can just ask it to "generate a javascript code that collects all the links on the page," but that does not work with the demo prompt on hugging face.

Does it allow training prompts or is that done through more fine tuning in this model?

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#125

Post by srush »

ComputerGuru wrote: post #122

Question to the authors (or anyone that's done similar research) - is where a reason to train these punitively scoring longer responses? Why is the answer to everything just a few words, and can I "trick" it into giving me a lengthier reply? (I tried "Give me a 200 word summary of ..." but that didn't help.)

We fine-tuned the model on a dozens of different NLP datasets and tasks in a prompted style. You can read all the prompts in the appendix or get them all here: https://github.com/bigscience-workshop/promptsource . Most NLP tasks are not particularly freeform, or they are naturally length limited like summary (XSum is very short). As a consequence, the model mostly defaults to short responses. Your "trick" is not that unreasonable though! Many of the training prompts that want long responses, ask for them explicitly.

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#126

Post by shantara »

Mizza wrote: post #12

The hosted demo has the default query, "How many hydrogen atoms are in a water molecule?" It said "two". I asked it, "How many oxygen atoms are in a water molecule?". It said "two".

>What is the square root of 1?

0.5

>How many oceans are there on Earth?

two

>Who was Juliette’s beloved?

Charles

>When did humans first land on the Moon?

July 1969

>How many sides are there in a rectangle?

Four

>How many sides are there in a circle?

Four

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#127

Post by ospohngellert »

ospohngellert wrote: post #123
gnarbarian wrote: post #94

Earlier quoted context omitted.

but the programmer is more likely to be a man, that's my point.

Yes, but the question is not whether that's true, but whether that's useful . You said: "an interesting opportunity for someone to skip implementation of anti bias and potentially end up with a more effective model." Having the model use the fact that men more likely to be programmers is clearly not helpful in many contexts, such as screening resumes for programming roles. In that context, it will cause the model to…

To add another example, say a model learned that ice cream sales correlate well to forest fire rates. Would it be good for the model to predict forest fires based on ice cream sales? The answer is no, because there is no causal link.

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#128

Post by petulla »

6gvONxR4sf7o wrote: post #112

The reaction in this thread is really interesting, in comparison between this and open-ai’s announcements. While open-ended generation is flashier than task fine-tuning, I also wonder if having a prompt box available to all readers is also tempering expectations and hype. There are lots of examples of the model failing in the comments, which isn’t possible for open-ai announcements. Having spent a ton of time with GP…

Great observation. Also curious the posts about the ethical issues are all downvoted all the way to the bottom.

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#129

Post by tiborsaas »

RyEgswuCsn wrote: post #77
tiborsaas wrote: post #59

Earlier quoted context omitted.

What's the difference? Answering a question can be considered "autocomplete".

Parent surely meant spellcheck autocompletion.

I doubt it, that's clearly exceeded by these language models. Calling it just an autocomplete - because it can mean a lot of things people are familiar with - is a way to downplay their significance.

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#130

Post by tibbar »

GPUboy wrote: post #124
stellaathena wrote: post #3

[Disclaimer: I am an author of the above paper and played a rather minimal role. I am also a prominent member of EleutherAI.] "Instruction-tuning" is clearly in the air. Simultaneous work at Google (released less than two weeks ago) on a model they call FLAN can be found here: https://ai.googleblog.com/2021/10/introducing-flan-more-gene... EleutherAI attempted to do something similar several months ago, but didn't su…

Thank you for this! Could you or anyone available please explain how to get it to generate javascript like with GPT-3? For example, with gpt-3 you can just ask it to "generate a javascript code that collects all the links on the page," but that does not work with the demo prompt on hugging face. Does it allow training prompts or is that done through more fine tuning in this model?

Code generation is not supported due to the tokenization strategy.
Post reply on HN