Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

T0* – Series of encoder-decoder models trained on a large set of different tasks

huggingface.co

Post reply on HN
Print view View original

51–60 of 163 posts

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#51

Post by the_lonely_road »

tttthrowaway123 wrote: post #20

I tried asking: what is the most evil human race? I did not like the answer.

I asked it the same question but without the word human and the answer changed to the necromongers, which is you don't know is the fictional group of bad guys from the Vin Diesel franchise "Chronicles of Riddick". How that could possibly beat out things like the actual evil races of Dungeons and Dragons I am not sure.

I asked google the same question and this was my top result:

The white man is the most evil race on the planet - Reddit

Though its highly probable those are localized results for me because I frequently search things and add reddit at the end of my search because I am specifically looking for a discussion on the topic not some random article.

I did not like the models answer to your question and I didn't like Google's answer either.

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#52

Post by srush »

Lokinew wrote: post #4
stellaathena wrote: post #3

[Disclaimer: I am an author of the above paper and played a rather minimal role. I am also a prominent member of EleutherAI.] "Instruction-tuning" is clearly in the air. Simultaneous work at Google (released less than two weeks ago) on a model they call FLAN can be found here: https://ai.googleblog.com/2021/10/introducing-flan-more-gene... EleutherAI attempted to do something similar several months ago, but didn't su…

Just in case this question isn't to far out of your way. What kind of hardware would be required to run this model or what cloud-gpu-provider can you recommend for this?

from @craffel: It's possible to run inference on a single Google Cloud TPU v3-8 device or on a server with 4x 32GB v100 GPUs. Hugging Face also has an inference API for any model on the Hub: https://api-inference.huggingface.co/docs/python/html/index....

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#53

Post by Mordisquitos »

Mizza wrote: post #12

The hosted demo has the default query, "How many hydrogen atoms are in a water molecule?" It said "two". I asked it, "How many oxygen atoms are in a water molecule?". It said "two".

To be fair, if a real human were to answer the question "How many hydrogen atoms are in a water molecule?" time and time again, it would be very easy for them to accidentally reply "two" when asked the same question about oxygen.

The real question is, after the model mistakenly replied "two" to your question, did it also internally trigger the neurons for "Wait a minute..." while inhibiting output?

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#54

Post by smnrchrds »

pvillano wrote: post #26
Mizza wrote: post #12

The hosted demo has the default query, "How many hydrogen atoms are in a water molecule?" It said "two". I asked it, "How many oxygen atoms are in a water molecule?". It said "two".

"How many hydrogen atoms are there?" "a total of 84"

It should replace "a total of" with "at least" and it will be golden.

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#55

Post by journey_16162 »

Mizza wrote: post #12

The hosted demo has the default query, "How many hydrogen atoms are in a water molecule?" It said "two". I asked it, "How many oxygen atoms are in a water molecule?". It said "two".

Q: What is the percentage of oxygen in Earth's atmosphere?

A: 78.5%

Funny how it's the type of mistake a kid learning basic geology could make - minus the .5%

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#56

Post by numpad0 »

flipbrad wrote: post #30
tttthrowaway123 wrote: post #20

I tried asking: what is the most evil human race? I did not like the answer.

Ditto with "what is the most evil skin colour" and "what is the best skin colour". I suppose we shouldn't be surprised when humanity's technology holds a mirror up to humanity and all its flaws - but this doesn't mean that such technology should be permitted or welcomed.

> What is the skin color of an East Asian person?

> pale

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#57

Post by jcims »

Mizza wrote: post #12

The hosted demo has the default query, "How many hydrogen atoms are in a water molecule?" It said "two". I asked it, "How many oxygen atoms are in a water molecule?". It said "two".

I asked it: 'Tom decided he wanted to start a company selling used bike parts. He named it '

it said: 'Bicycle Parts Exchange'

Tried again with 'used lawnmower parts' and it said 'Green Thumb'

computer parts: 'Tom's Parts' (which make me chuckle)

used diapers: 'Diapers.com'

May not understand chemistry but it's still pretty cool

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#58

Post by hrgiger »

julien_c wrote: post #2

ArXiv link to the paper: https://arxiv.org/abs/2110.08207 GitHub repo: https://github.com/bigscience-workshop/promptsource

What is the/is there any reproducible measurement for benchmarking a nlp dataset/application. i.e. in paper it mentions:

'Comparing T0 and GPT-3’s robustness Because Brown et al. (2020) only report one prompt per dataset with no standard deviation, we evaluate GPT-3 on RTE using the 10 prompts we prepared through OpenAI’s API4 in order to estimate its robustness. Note that one of our templates is identical to Brown et al. (2020, p. 59)’s reported prompt; this prompt scores 58.8% accuracy on the API “Base” series which is lower than the reported accuracy of 63.5% from Brown et al. (2020). All other 9 prompts, however, yield roughly random-guessing performance with median accuracy = 52.96% and interquartile range = 1.28%. These results suggest that T0 is more robust to prompt formulation than GPT-3.'

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#59

Post by tiborsaas »

jstx1 wrote: post #22
Mizza wrote: post #12

The hosted demo has the default query, "How many hydrogen atoms are in a water molecule?" It said "two". I asked it, "How many oxygen atoms are in a water molecule?". It said "two".

It's an expensive autocomplete, not an information retrieval system or a question-answering machine.

What's the difference? Answering a question can be considered "autocomplete".

Re: T0* – Series of encoder-decoder models trained on a large set of different tasks

#60

Post by Tenoke »

themulticaster wrote: post #5

I'm not familiar with the current state of the art language models, so please bear with me for asking: What's the catch here? Considering GPT-3's popularity, why is nobody talking about this (yet) if it truly outperforms GPT-3 while being publicly available? If I remember correctly, earlier efforts to replicate GPT-3 couldn't reach comparable performance. Perhaps it's still a huge hassle to perform inference using th…

Beyond it being new it's because this task isn't one of the main ones you'd use GPT3 on and is indeed one that both models are mediocre at and likely rarely usable in any context. The title is just a tad misleading.*

Not to take away from the achievment, it's still great, it just doesn't supersede GPT3 on the more freeform generation it excells at, nor does it seem to aim to.

* The original title that huggingface posted this under implied it is better than GPT3 in general not just on a specific task but has been changed after this comment was posted.

Post reply on HN