Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

Post by ars » Fri, Oct 25, 2019, 2:39 AM UTC

briga wrote: post #54

mannykannot wrote: post #50

Earlier quoted context omitted.

From the abstract of the associated paper: "performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research." It occured to me that hn_throwaway_99's question, and the responses to it, is the sort of dialog in which one could find additional headroom for further research into natural language understanding. We can understand, for example, that while…

Limited headroom? Seems like they're assuming greater-than-human language ability is just impossible and will never be surpassed.

Post by chrisweekly » Fri, Oct 25, 2019, 2:46 AM UTC

Enginerrrd wrote: post #48

hn_throwaway_99 wrote: post #15

Earlier quoted context omitted.

Ahh, just found an example where that's taken from https://glosbe.com/en/en/land . If you find on that page you'll see the exact sentence "the enemy landed several of our aircraft" (without the s after aircraft) which it says means "shoot down". I have still never heard landed used in that way, and again in other dictionaries I searched I couldn't find that definition either. Thus, this is a case where the "AI" may g…

I think if we really looked at it, it likely comes from fishing where "to land" a fish means to succeed in quite literally getting it onto land from the water. But we use it as "to successfully get" (something typically uncertain) in many other contexts.

Post by mannykannot » Fri, Oct 25, 2019, 2:51 AM UTC

briga wrote: post #54

mannykannot wrote: post #50

Earlier quoted context omitted.

From the abstract of the associated paper: "performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research." It occured to me that hn_throwaway_99's question, and the responses to it, is the sort of dialog in which one could find additional headroom for further research into natural language understanding. We can understand, for example, that while…

Limited headroom? Seems like they're assuming greater-than-human language ability is just impossible and will never be surpassed.

Post by calabin » Fri, Oct 25, 2019, 2:53 AM UTC

throwaway_bad wrote: post #6

Possibly dumb question: How do you ensure there's no data leakage when benchmarking transfer learning techniques? Is that even a problem anymore when the whole point is to learn "common sense" knowledge? For example their “Colossal Clean Crawled Corpus” (C4), a dataset consisting of hundreds of gigabytes of clean English text scraped from the web, might contain much of the same information as the benchmark datasets,…

Post by 6gvONxR4sf7o » Fri, Oct 25, 2019, 2:57 AM UTC

gradys wrote: post #16

pmoriarty wrote: post #4

There was an article[1] posted to HN recently about these benchmarks, and it was pretty skeptical. Regarding SuperGLUE specifically, it asked: "Indeed, Bowman and his collaborators recently introduced a test called SuperGLUE that's specifically designed to be hard for BERT-based systems. So far, no neural network can beat human performance on it. But even if (or when) it happens, does it mean that machines can really…

This feels hollow. Can't this be said about any benchmark? It seems natural and proper that as one benchmark becomes saturated, we introduce harder benchmarks. I don't think anyone in the field thinks that once we match human performance on benchmark X, we're officially done. It just means it's time for more interesting benchmarks. Over time, if it starts to become difficult to design benchmarks that humans can outpe…