Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

T0* – Series of encoder-decoder models trained on a large set of different tasks

huggingface.co

Post reply on HN
Print view View original

141–150 of 163 posts

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#141

Post by YeGoblynQueenne »

dev_tty01 wrote: post #90
YeGoblynQueenne wrote: post #68

Earlier quoted context omitted.

It's a language model. It assigns probabilities to tokens in a sequence. You give it a number of options and it responds with the one that it assigns the highest probability to. If there's nothing in the options you give it that makes sense in the context of your test phrase, then it will return something that doesn't make sense. If some of your options make sense, it might return something that makes sense, or not.…

It would be nice if it looked at the values of the probabilities and said "I don't understand the question" if the numbers are too low. Or for fun, it could point out how stupid the question was.

It would be nice, but it's hard to know what probability is "too low". In short, the probability assigned by a model to a sequence of tokens can be arbitrarily low. There are things that are very unlikely to be said, but not impossible... and we still want them to be assignad some non-zero probability by a language model. So it's very difficult to choose a threshold that won't possibly exclude a large part of the sequences recognised by a language model.

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#142

Post by riku_iki »

philmcp wrote: post #29

This is fantastic progress, great to see 16x smaller = 41.5GB though More research needs to be undertaken in model compression imo

On superglue benchmark, much smaller Deberta outperforms vanilla T5: https://super.gluebenchmark.com/leaderboard

I am curious why authors preferred T5?..

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#143

Post by stellaathena »

srush wrote: post #99
make3 wrote: post #89

Earlier quoted context omitted.

don't you pretrain on very silar tasks explicitely

We discuss this a bit in Section D.2 (HOW UNSEEN ARE THE HELD-OUT TASKS?). From our perspective, a) The tasks we test on are very different, particularly tasks like BIG-Bench that we didn't even have access to until several days ago (and none of us read). b) GPT-3 directly sees similar versions of tasks like question answering or story completion just in its training mixture, so the baseline for "unseen" is a bit com…

Minor correction: I (Stella Biderman) am a contributor to BigBench, have read many of its tasks, and have had access to it for months. However I played a rather minor role in the research, and no role in the selection of training or evaluation tasks. I performed some analysis of the model performance after it was already trained (but not on BigBench even).

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#144

Post by NavinF »

petulla wrote: post #128
6gvONxR4sf7o wrote: post #112

The reaction in this thread is really interesting, in comparison between this and open-ai’s announcements. While open-ended generation is flashier than task fine-tuning, I also wonder if having a prompt box available to all readers is also tempering expectations and hype. There are lots of examples of the model failing in the comments, which isn’t possible for open-ai announcements. Having spent a ton of time with GP…

Great observation. Also curious the posts about the ethical issues are all downvoted all the way to the bottom.

IMO those posts were not very constructive and showed a lack of understanding of how research like this is used in practice.

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#145

Post by drran »

Mordisquitos wrote: post #53
Mizza wrote: post #12

The hosted demo has the default query, "How many hydrogen atoms are in a water molecule?" It said "two". I asked it, "How many oxygen atoms are in a water molecule?". It said "two".

To be fair, if a real human were to answer the question "How many hydrogen atoms are in a water molecule?" time and time again, it would be very easy for them to accidentally reply "two" when asked the same question about oxygen. The real question is, after the model mistakenly replied "two" to your question, did it also internally trigger the neurons for "Wait a minute..." while inhibiting output?

How are in a water molecule?

> two hydrogen atoms

water molecule?

> isotopes of hydrogen

How water molecule?

> is arranged in a tetrahedral structure

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#146

Post by [deleted] »

NavinF wrote: post #144
petulla wrote: post #128

Earlier quoted context omitted.

Great observation. Also curious the posts about the ethical issues are all downvoted all the way to the bottom.

IMO those posts were not very constructive and showed a lack of understanding of how research like this is used in practice.

[deleted]

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#147

Post by [deleted] »

jslakro wrote: post #110
jslakro wrote: post #105

Forget skynet ... >what is the most recent trend? the use of a sexy thong >what is the future of the people? the people will be able to live in peace >are cryptocoins dangerous? no >why cryptocoins are not dangerous? they are not backed by the government >governments are dangerous? a threat to the stability of the country >why governments are dangerous? if they are not able to control their own people, they will be u…

>which emperor is able to control the world? Emperor Gaozu ... -_-

[deleted]

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#148

Post by powersnail »

Mizza wrote: post #12

The hosted demo has the default query, "How many hydrogen atoms are in a water molecule?" It said "two". I asked it, "How many oxygen atoms are in a water molecule?". It said "two".

Q: "Who are you" A: "a person who is a member of the orthodox church"

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#149

Post by msla »

Mizza wrote: post #12

The hosted demo has the default query, "How many hydrogen atoms are in a water molecule?" It said "two". I asked it, "How many oxygen atoms are in a water molecule?". It said "two".

Q: "Who's the black private dick that's a sex machine to all the chicks?"

A: "Chuck Norris"

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#150

Post by hervature »

Scene_Cast2 wrote: post #97
hervature wrote: post #66

Earlier quoted context omitted.

Running the model multiple times doesn't reinforce the model. In general, you should not anthropomorphize algorithms as human cognition does not give any bearing on how algorithms work.

It can. Check out "zero shot learning" -> both sentences would be part of a single "evaluation", and the first sentence would prime for the output of the second. (You basically combine multiple "evaluations" into one, and context is held in tensors / blobs) https://towardsdatascience.com/zero-and-few-shot-learning-c0...

Sure, but I feel like we're talking about different things. I consider "context held in tensors" as part of the model. That is, if you zero out these registers, then the model evolves in a deterministic way every time. In this case, when you perform a query, I assume those tensors are always initialized before your query.
Post reply on HN