Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

super.gluebenchmark.com

Post reply on HN
Print view View original

201–210 of 246 posts

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#201

Post by treis »

Accacin wrote: post #150
topspin wrote: post #127

Earlier quoted context omitted.

Would a native English speaker use the word "landed" in this way? In the context of aircraft? "Landed" is badly ambiguous here and several distinct meanings are plausible. Captured is the most natural word given your interpretation. Honestly that sentence -- the use of landed and that awful plural -- approaches engrish. Is that deliberate or is the use of English here just badly flawed? I can't see any other possibil…

I don't know. I guess I understood the sentence with 'landed' the same as I would have if someone told me that they'd 'landed a big job'. I wouldn't really say this myself though, although I hear people say 'landed a big catch' when they're talking about fishing.

Landed, with this meaning, is used in the context of successfully enticing someone to give you something. Like hooking a fish with bait.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#202

Post by dmurray »

hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

It's being used by analogy with "landing a fish". I've never heard it either, but I could believe it's in the argot of military airmen in some English-speaking country.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#203

Post by penagwin »

vagab0nd wrote: post #198

This is cool. Since they released a 11B pre-trained model, can we finally reproduce "unicorn-level" text generation now?

My understanding is that a lot of these really high performance models that reach for every percentage-point possible require an absurd amount of hardware - specifically an absurd amount of GPU memory.

For example I have what I consider a fairly "high end" rig for being a hobbyist individual, with 32GB of RAM, i7 8700k, 1080ti - there's 0 chance their model would fit on my system.

So I mean maybe if you have a ton of money? Usually what happens is a slimmer model with not "quite" as high of a score gets released that actually fits on consumer hardware.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#204

Post by OrangeMango »

nebulous1 wrote: post #186
_II__II_ wrote: post #57

Earlier quoted context omitted.

The example directly below that: "Justify the margins" and "The end justifies the means" is the one I find dubious. Obviously the former could mean to format a document, but those exact words in that structure could be a demand for someone to justify a financial margin for example. It is both true and false depending on the context.

I'm guessing this is intentional. To a human, although this could be somebody being asked to justify their financial margins that's not a very likely answer. The human can easily see that, while it's possible they're the same meaning, given the lack of any other context the answer is that they're not. The enemy could have landed several of our aircraft on one of their runways. Agassi may have beaten Becker over the h…

> The enemy could have landed several of our aircraft on one of their runways.

This is something that actually does happen. Less than 10 or 20 years ago, China did it to an US Air Force reconnaissance aircraft.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#205

Post by hn_throwaway_99 »

_II__II_ wrote: post #57
hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

The example directly below that: "Justify the margins" and "The end justifies the means" is the one I find dubious. Obviously the former could mean to format a document, but those exact words in that structure could be a demand for someone to justify a financial margin for example. It is both true and false depending on the context.

This is a good point I hadn't thought of. Honestly, I'm really not surprised anymore that the humans only scored 89%.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#206

Post by ArtWomb »

"Attention is all you need", indeed. Of course, our instinct tells us there is more to language inference than word proximity. And so results approaching or exceeding expert-level human baseline raise more questions than providing cause for popping champagne corks.

In Question Answering, which is also advancing rapidly with insights from transformers and denoising auto-encoders, but still far from human baseline. The ease with which these models can answer a sample question such as: "Who was the first human in space", demonstrates both their efficacy and limitations. Pre-trained on a large corpus of text, almost every document that contains the the name "Yuri Gagarin" will in its near vicinity describe him in relation to his pioneering accomplishment for which he became a cultural icon.

And for even more generalizable scenarios, such as "what might you find on a Mayan monument"? It becomes imperative that an agent explain its reasoning in natural language as well to enable self-correcting backpropagation of error correction.

Language may be considered low-dimensional relatively speaking. And sentence prediction across quotidian tasks manageable in current state-of-the-art architectures. But looking at how difficult it is to predict the next N frames of video given a short input example demonstrates the intractability of the problem in higher dimensional spaces.

Neural Models for Speech and Language: Successes, Challenges, and the Relationship to COmputational Models of the Brain - Michael Collins

https://www.youtube.com/watch?v=HVnFKmPaU8c

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#207

Post by saalweachter »

throwaway936482 wrote: post #143
hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

I think this is really interesting, because "the enemy landed several of our aircraft(s)" is the sort of sentence I'd have hauled a student up for using as a teacher, because 1) it's a none standard, arguably incorrect usage they've used either because they're a none native speaker or because they're trying to be clever and failing, and 2) because the plural of aircraft is aircraft. Nevertheless the author of this se…

As someone who spends a lot of time puzzling out intent, I would infer they are using "landed" to mean "grounded" in that context.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#208

Post by vagab0nd »

penagwin wrote: post #203
vagab0nd wrote: post #198

This is cool. Since they released a 11B pre-trained model, can we finally reproduce "unicorn-level" text generation now?

My understanding is that a lot of these really high performance models that reach for every percentage-point possible require an absurd amount of hardware - specifically an absurd amount of GPU memory. For example I have what I consider a fairly "high end" rig for being a hobbyist individual, with 32GB of RAM, i7 8700k, 1080ti - there's 0 chance their model would fit on my system. So I mean maybe if you have a ton of…

Maybe I'm oversimplifying, but it seems to me that once you have the model trained, it should be possible to partition it somehow when inferencing, to fit smaller machines. At least for a proof of concept it should be possible.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#209

Post by saalweachter »

QuinnWilton wrote: post #90
andrewstuart2 wrote: post #64

Earlier quoted context omitted.

One of my favorite examples that I heard in a David Rock talk which I can no longer find on youtube: "Time flies like an arrow": Time moves swiftly and in one direction. Record the speed of flies in the same way you would an arrow. Time flies, which are a kind of fly, are fond of an arrow. (e.g. Time flies like an arrow, fruit flies like a banana).

It sounds like you're talking about garden-path sentences [0], and in particular: "time flies like an arrow; fruit flies like a banana" [1]. These are sentences whose structure tricks the reader into making an incorrect parse. My favourite of these has always been: "The horse raced past the barn fell". [0] https://en.wikipedia.org/wiki/Garden-path_sentence [1] https://en.wikipedia.org/wiki/Time_flies_like_an_arrow;_f…

"The horse raced past the barn fell, which has been haunted since all those teenagers were murdered there."

(Noun-adjective is a rare formation, but amusingly more common in the same situations where the author uses rare and archaic definitions like the adjective "fell".)

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#210

Post by saalweachter »

tombh wrote: post #149
bllguo wrote: post #136

Earlier quoted context omitted.

I wouldn't use it that way myself, but at the same time the intended meaning is clear as day to me from the context. I'm surprised by the reactions. "Enemy" should give it away immediately.

I'm surprised too. This algorithm is about understanding language, and surely that includes understanding the intended usage . This is something humans have to do all the time. So what if there isn't a formally archived consensus on the definition of "landed" as used in the example. The intended meaning is clear, and so hats off to the algorithm for rolling with it, that is in my mind the fundamental goal of understa…

It's more or less impressive depending on whether the algorithm already ate a dictionary; then it's the difference between inferring from context, as people do, and simply knowing all of the known unconventional usages in a very inhuman way.
Post reply on HN