Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

super.gluebenchmark.com

Post reply on HN
Print view View original

71–80 of 246 posts

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#71

Post by ethbro »

natch wrote: post #59
hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

So many options for sentence number two. - The enemy stole the aircrafts, and after some drama in flight managed to land several of them. - The enemy used remote control to force them to land. - The enemy used coercive force to force our pilots to land them. - The enemy captured them. - The enemy shot them down. - During a friendly event while we set our differences with our enemy aside and agreed to fly each other's…

- An ML algorithm from the future traveled back in time, writing specific SuperGLUE examples to poison AI research, thereby preventing the emergence of a competitive AI which would also master the secrets of closed timelike curves

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#72

Post by adrianmonk »

mike00632 wrote: post #20
hn_throwaway_99 wrote: post #15

Earlier quoted context omitted.

Ahh, just found an example where that's taken from https://glosbe.com/en/en/land . If you find on that page you'll see the exact sentence "the enemy landed several of our aircraft" (without the s after aircraft) which it says means "shoot down". I have still never heard landed used in that way, and again in other dictionaries I searched I couldn't find that definition either. Thus, this is a case where the "AI" may g…

I've worked in aviation for 8 years and also didn't understand this use of "landed". I've heard "grounded" used like this: "The maintenance issues gounded the jet," but not "landed".

Working in aviation probably puts you in a mindset that makes it harder to parse. It's not being used in a way that is related to flight or aircraft.

It's like if people were discussing where to have a conference, and one of them proposed a hotel. Then another person suggested a resort. Then a third person floated a cruise ship. Cruise ships do float, but it has nothing to do with anything. They are floating the idea of the ship as a venue.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#73

Post by jcims »

6gvONxR4sf7o wrote: post #58

One thing to always point out in these cases is that the human baseline isn't "how well people do at this task," like it's often hyped to be. It's "how well does a person quickly and repetitively doing this do, on average." The 'quickly and repetitively' part is important because we all make more boneheaded errors in this scenario. The 'on average' part is important because the errors the algo makes aren't just fewer…

Great point! It makes sense in the context of what these algorithms would generally be tasked with.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#74

Post by jcims »

throwaway_bad wrote: post #6

Possibly dumb question: How do you ensure there's no data leakage when benchmarking transfer learning techniques? Is that even a problem anymore when the whole point is to learn "common sense" knowledge? For example their “Colossal Clean Crawled Corpus” (C4), a dataset consisting of hundreds of gigabytes of clean English text scraped from the web, might contain much of the same information as the benchmark datasets,…

Not dumb at all and probably a major challenge when developing benchmarks.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#75

Post by jcims »

hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

In looking through many of the replies to this downstream, it appears that the system is actually correct in that there's an obscure use of 'land' at play in the second sentence.

It makes me think that there's going to be many adversarial examples of text that humans parse one way because of common usage while machines parse another way because of details like this.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#76

Post by rhizome »

hn_throwaway_99 wrote: post #15
matthewowen wrote: post #7

Earlier quoted context omitted.

I think it's landed in the same sense as "landed a deal": got, or achieved, in this case achieving shooting them down. For me, my first read of the sentence would definitely be that it means shot down.

Ahh, just found an example where that's taken from https://glosbe.com/en/en/land . If you find on that page you'll see the exact sentence "the enemy landed several of our aircraft" (without the s after aircraft) which it says means "shoot down". I have still never heard landed used in that way, and again in other dictionaries I searched I couldn't find that definition either. Thus, this is a case where the "AI" may g…

>If you find on that page you'll see the exact sentence "the enemy landed several of our aircraft" (without the s after aircraft) which it says means "shoot down".

They're not shy about illustrating a military application up front!

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#77

Post by brainless »

hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

I think the difference in these sentences is about the way to land. In sentence 1, the pilot of the aircraft is in control. In sentence 2, the pilots are not in control, the enemy forced them to land (whatever the means).

If I read these two sentences in context of some news, they would evoke very different "landing" scenes in my head.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#78

Post by changoplatanero »

andrewstuart2 wrote: post #64
_II__II_ wrote: post #57

Earlier quoted context omitted.

The example directly below that: "Justify the margins" and "The end justifies the means" is the one I find dubious. Obviously the former could mean to format a document, but those exact words in that structure could be a demand for someone to justify a financial margin for example. It is both true and false depending on the context.

One of my favorite examples that I heard in a David Rock talk which I can no longer find on youtube: "Time flies like an arrow": Time moves swiftly and in one direction. Record the speed of flies in the same way you would an arrow. Time flies, which are a kind of fly, are fond of an arrow. (e.g. Time flies like an arrow, fruit flies like a banana).

"I eat my rice with butter." could mean that you use butter as a utensil to eat your rice with. There is often an unlikely way of parsing the sentence that gives an alternate meaning. The point is to test the computer to see if it can distinguish the likely parse from an unlikely one.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#79

Post by gvhst »

hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

It sounds like the “landed” in 2. is similar in usage as “landed” used in the turn of phrase “he landed the deal.”

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#80

Post by solotronics »

ethbro wrote: post #71
natch wrote: post #59

Earlier quoted context omitted.

So many options for sentence number two. - The enemy stole the aircrafts, and after some drama in flight managed to land several of them. - The enemy used remote control to force them to land. - The enemy used coercive force to force our pilots to land them. - The enemy captured them. - The enemy shot them down. - During a friendly event while we set our differences with our enemy aside and agreed to fly each other's…

- An ML algorithm from the future traveled back in time, writing specific SuperGLUE examples to poison AI research, thereby preventing the emergence of a competitive AI which would also master the secrets of closed timelike curves

Actually the algo was able to determine we exist in a simulation and perform meta programming by hacking the sim infrastructure (higher order dimensions of spacetime) and rewriting the future which to us appears that it traveled to the past.
Post reply on HN