Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

super.gluebenchmark.com

Post reply on HN
Print view View original

211–220 of 246 posts

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#211

Post by main5tream »

dmurray wrote: post #202
hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

It's being used by analogy with "landing a fish". I've never heard it either, but I could believe it's in the argot of military airmen in some English-speaking country.

It's conceptually the same - having an entity go from water or air to the the ground. The hard part would be to associate the fact that there's no way for an 'enemy' to land the aircraft other than to do so forcibly which implies shooting it down.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#212

Post by craffel »

mannykannot wrote: post #177
craffel wrote: post #98

Earlier quoted context omitted.

Hi, one of the paper's authors here. We didn't submit our model's predictions for the AX-b task yet, we just copied over the predictions from the example submission. We will submit predictions for AX-b in the next few days.

RcouF1uZ4gsC makes a compelling case for the results on this test to potentially be a significant caveat to the results, and also to the claims of achieving a near-human level of performance. If so, then why would you make such claims before you have these results? Or at least mention this caveat at the points where you are making the claim, such as in the abstract.

To be clear, here is the claim we make in the paper (we did not write the title of this post to HN):

> For SuperGLUE, we improved upon the state-of-the-art by a large margin (from an average score of 84.6 [Liu et al., 2019c] to 88.9). SuperGLUE was designed to comprise of tasks that were “beyond the scope of current state-of-the-art systems, but solvable by most college-educated English speakers” [Wang et al., 2019b]. We nearly match the human performance of 89.8 [Wang et al., 2019b]. Interestingly, on the reading comprehension tasks (MultiRC and ReCoRD) we exceed human performance by a large margin, suggesting the evaluation metrics used for these tasks may be biased towards machine-made predictions. On the other hand, humans achieve 100% accuracy on both COPA and WSC, which is significantly better than our model’s performance. This suggests that there remain linguistic tasks that are hard for our model to perfect, particularly in the low-resource setting.

I'm not sure why the SuperGLUE/GLUE benchmark was designed to omit the AX-* scores from the benchmark score. It may be that they have no corresponding training set.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#213

Post by mcabbott »

rjvs wrote: post #133
topspin wrote: post #127

Earlier quoted context omitted.

Would a native English speaker use the word "landed" in this way? In the context of aircraft? "Landed" is badly ambiguous here and several distinct meanings are plausible. Captured is the most natural word given your interpretation. Honestly that sentence -- the use of landed and that awful plural -- approaches engrish. Is that deliberate or is the use of English here just badly flawed? I can't see any other possibil…

There are a lot of native English speakers in the world and not all of them use the same idioms that you do. This seems like perfectly valid English to me; some other words that could be used instead of “landed” in the aircraft sentence include “bagged”, “nabbed”, “poached”, “got” and “did in”. One of the entertaining aspects of English is the multitude of ways it can be used.

Those are all good synonyms for "got" in the context of shooting at things. But none of the others already has a strong meaning in the context of aircraft, and this other meaning does create some confusion, which is why many speakers would avoid it (if thinking clearly).

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#215

Post by craffel »

throwaway_bad wrote: post #105
craffel wrote: post #104

Earlier quoted context omitted.

Hi, one of the paper authors here. Indeed this is a good question. A couple of comments: - Common Crawl overall is a sparse web dump, it is unlikely that the month we used includes any of the data that are in any of the test sets. - In order for the data to be useful to our model, it would have to be in the correct preprocessed format. ("mnli: hypothesis: ... premise: ...") with the label in a format our model could…

> Big neural nets trained with SGD are unlikely to memorize something if they only see it once over the course of one million training steps I am not so sure about that. Have you seen this thread: https://www.reddit.com/r/MachineLearning/comments/dfky70/dis... Apparently lots of sentence fragments were memorized in GPT-2 (including real world URLs, entire conversations, username/emails and other PII).

It actually can be more pernicious than that: https://arxiv.org/abs/1802.08232

However note that the dataset used to train GPT-2 is about 20x smaller than C4. I'm not 100% sure how many times the training set was repeated over the course of training for GPT-2, but it was likely many times. I stand by my statement (that memorization is unlikely with SGD and no repetition of training data) but I would be happy to be proven otherwise.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#216

Post by YeGoblynQueenne »

nopinsight wrote: post #97

As someone working in the field, I congratulate the excellent accomplishment but agree with the authors that we shouldn't get too excited yet (their quote below after the four reasons). Here are some reasons: 1) Most likely, the model is still susceptible to adversarial triggers as demonstrated on other systems here: http://www.ericswallace.com/triggers 2) T5 was trained with ~750GB of texts or ~150 billion words, wh…

I think that the point about the majority of tests being multiple-choice is the most important one to underline.

Structuring a problem as a multiple choice task is basically turning it into a classification problem, but it doesn't really answer the question everyone wants answered: is it really possible to reduce the problem of language understanding to classification? i.e. is it really possible to understand human language with no other ability than the ability to identify the classes of objects?

But that is a question that has to be answered before any performance on benchmarks that reduce language understanding to classification can be appraised correctly. If accurate classification is not sufficient for language understanding, then beating benchmarks like SuperGLUE tells us nothing new (we already know we have good classifiers).

The problem here is that we have no good measures of language understanding, of humans or machines- because we have a poor, er, understanding of our own language ability. Until we know more about what it means to understand language it won't be possible to evaluate automated language understanding systems very well.

Hopefully though, the skepticism I've observed around results like the one above, will lead to a renewed effort to research our language ability, and perhaps our intelligence in general.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#217

Post by jasonlotito »

hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

So, here is the thing. ML shouldn't just be about learning rules. It should be about actually learning, and understanding.

Just because you've never heard the word used that way, you were able to infer it meant something different. Even with the use of aircrafts.

We all make mistakes when writing or speaking. We don't let that get in the way of interpreting the information being passed. Even if we post comments that contain errors.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#218

Post by madis »

rendall wrote: post #188
HereBeBeasties wrote: post #164

Earlier quoted context omitted.

If you teach others English, please learn the difference between "none" and "non". You mean "non-standard" in all your examples here (if British) or perhaps "nonstandard" (if American).

Sssh, pumpkin. We live in a world of autocorract.

Sssh dysgraphic. It's not an excuse.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#219

Post by jasonlotito »

alex_young wrote: post #154
hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

The plural of aircraft is aircraft. Not aircrafts. https://www.grammar-monster.com/plurals/plural_of_aircraft.h... Maybe the _examples_ for a language test should be grammatically correct?

Maybe the examples for a language test should use language that people actually use every day.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#220

Post by YeGoblynQueenne »

YeGoblynQueenne wrote: post #85

Assuming that the baseline human score was set according to the performance of adult humans, then according to these results T5 has a language understanding ability at least as accurate as a human child. In fact it's not just T5 that should be able to understand language as well as a human child, but also BERT++, BERT-mtl and RoBERTa, each of which has a score of 70 or more. There really shouldn't be anything else on…

To clarify, I meant this comment as an expression of skepticism- I don't believe that the SuperGLUE benchmark really evaluates language understanding, or that BERT and friends are within a few percents of human language understanding. I think SuperGLUE is just another benchmark that is measuring something else than what it's supposed to be measuring (machine learning benchmarks usually do).

It seems that the teams behind the attempts to beat such benchmarks are aware of the weaknesses of the benchmarks though, so that's encouraging.

Post reply on HN