Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

super.gluebenchmark.com

Post reply on HN
Print view View original

31–40 of 246 posts

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#31

Post by taneq »

throwaway_bad wrote: post #6

Possibly dumb question: How do you ensure there's no data leakage when benchmarking transfer learning techniques? Is that even a problem anymore when the whole point is to learn "common sense" knowledge? For example their “Colossal Clean Crawled Corpus” (C4), a dataset consisting of hundreds of gigabytes of clean English text scraped from the web, might contain much of the same information as the benchmark datasets,…

You don't. Even humans frequently leak information like this. It's just a consequence of having incomplete or incompletely analyzed information.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#32

Post by ehsankia »

lettergram wrote: post #25

Although those are some great results, I wish I could try it out locally... https://github.com/google-research/text-to-text-transfer-tra... It drives me nuts that most of these papers / publications don't have code where I can just run: > python evaluate_model.py Still exciting, just annoying that I'd have to set up google cloud to try this out.

They often do setup Python notebooks / Colabs you can simply run, especially with the data hosted on GCloud. Unfortunately not this time.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#33

Post by tanilama »

hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

It means 'succeeded in shooting down' right? Seems pretty contextual, but understandable.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#34

Post by didibus »

hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

It's weird, because I understood the second one as meaning shoot down, yet to me that's the same definition of landed. You just assume the enemy didn't land them gracefully without a scratch, because they are well, enemies.

So I would have answered that the word meant the same thing.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#37

Post by fouc »

hn_throwaway_99 wrote: post #15
matthewowen wrote: post #7

Earlier quoted context omitted.

I think it's landed in the same sense as "landed a deal": got, or achieved, in this case achieving shooting them down. For me, my first read of the sentence would definitely be that it means shot down.

Ahh, just found an example where that's taken from https://glosbe.com/en/en/land . If you find on that page you'll see the exact sentence "the enemy landed several of our aircraft" (without the s after aircraft) which it says means "shoot down". I have still never heard landed used in that way, and again in other dictionaries I searched I couldn't find that definition either. Thus, this is a case where the "AI" may g…

I agree, AI should realistically be able to detect the rare/uncommon/ambiguous usage as well, and rated for that.

I suppose in some case it could score better than humans on SuperGLUE benchmark.. but eventually it will have to come back down to near human score as it gets more accurate.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#39

Post by vonseel »

tanilama wrote: post #33
hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

It means 'succeeded in shooting down' right? Seems pretty contextual, but understandable.

Seems like a really odd way of saying it but that’s what I’d think too, as in “landed their shots”.

This is either a poor question, or a really great question, if the goal of the test is to confuse computers where a human would normally say “huh, weird way of saying that but I guess they mean...”.

Post reply on HN