Live data from Hacker News

Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

super.gluebenchmark.com

231–240 of 246 posts

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#231

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

Well "he landed the deal" implies a score or a hit. So to say they "landed" the planes could vaguely make sense but it is hardly good English. They might have been thinking of "grounded"?

'Grounded' means the plane could not take off. It was on the ground and must remain there.

Landing a deal (or a fish) is like landing a plane. A human acts to cause a desired outcome. Unlike forcing a pilot to involuntarily land a plane, the perspective of the fish as involuntarily being forced to land is not a necessary inference for this use of 'land'.

Geez, language can be subtle.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#232

As someone working in the field, I congratulate the excellent accomplishment but agree with the authors that we shouldn't get too excited yet (their quote below after the four reasons). Here are some reasons: 1) Most likely, the model is still susceptible to adversarial triggers as demonstrated on other systems here: http://www.ericswallace.com/triggers 2) T5 was trained with ~750GB of texts or ~150 billion words, wh…

> 1) Most likely, the model is still susceptible to adversarial triggers as demonstrated on other systems here Humans are susceptible to adversarial triggers too, so this doesn't necessarily make the model less impressive. It is a big problem in practical use though.

I am curious on what you mean by adversarial examples/triggers for humans in the domain of natural language.

Off the top of my head, I can think of:

* garden path sentences

* highly recursive sentences

Could you or anyone provide some other classes?

The two classes above however can generally be understood by a large number of educated native speakers with time to think carefully.

Humans also do not get derailed so badly as in the examples in this link. http://www.ericswallace.com/triggers

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#233
post #201

Earlier quoted context omitted.

I don't know. I guess I understood the sentence with 'landed' the same as I would have if someone told me that they'd 'landed a big job'. I wouldn't really say this myself though, although I hear people say 'landed a big catch' when they're talking about fishing.

Landed, with this meaning, is used in the context of successfully enticing someone to give you something. Like hooking a fish with bait.

FWIW, Landing a fish is not the same as hooking it. Landing a fish literally means pulling it to land (or boat).

So landing=catching=scoring.

Depending on the type of fishing, you can still be an underdog to land the fish after hooking it.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#234

Earlier quoted context omitted.

I think this is really interesting, because "the enemy landed several of our aircraft(s)" is the sort of sentence I'd have hauled a student up for using as a teacher, because 1) it's a none standard, arguably incorrect usage they've used either because they're a none native speaker or because they're trying to be clever and failing, and 2) because the plural of aircraft is aircraft. Nevertheless the author of this se…

As someone who spends a lot of time puzzling out intent, I would infer they are using "landed" to mean "grounded" in that context.

I would have assumed the second used the term landed to mean acquired. But only after being told that it’s meaning is supposed to be different from the first. With no other context from those two sentences, I’d have guessed #2 Meant land the same way as #1

One other point: I’ve never heard the term “landed” to mean “grounded”, which is maybe the actual intent of #2, but maybe the ai sentence generation is off.....

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#235
post #117
post #96

Earlier quoted context omitted.

My mother got a perfect 800 score on the GRE English test many years ago when she wanted to go back to graduate school after her children were grown up enough (highschool/college age). She told me that the way she got her perfect score was by realizing when the questions were wrong and thinking of what answer the test creators believed to be correct. She had to outguess the test creators and answer the questions wron…

I've had the 'pleasure' of taking some 'Microsoft certifications' at various companies I worked at in the past and this sounds extremely familiar. "I probably won't ever do it like that and/or there's a syntax error in all four of the answers... but this is the answer you want to hear. It's wrong, mind you, but it's what you want to hear."

Reminds me of the 1 question I got "wrong" on a DOS test (years ago) at TAFE.

The question was "How do you delete all files in the current directory?". Using DOS 6.22 (I think, it's from memory).

My answer "del." was marked incorrect. Because the teacher didn't know enough about DOS to understand that's the standard shortcut for "del .". And the teacher refused to even try out the command, lets alone fix the incorrect mark. sigh

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#236

Earlier quoted context omitted.

2. Reminds me a of a theory that Iran landed an American stealth drone by sending spoofed signals.

Still ambiguous. Landed as in make it contact the ground or landed as in obtain, like in landing a job? For me taking an airborne object and making it touch the ground is pretty much the same meaning whether it's from the inside or remotely or shooting it down.

Yes. I think "ambiguous" is the best word to describe all of this.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#237

Earlier quoted context omitted.

> 1) Most likely, the model is still susceptible to adversarial triggers as demonstrated on other systems here Humans are susceptible to adversarial triggers too, so this doesn't necessarily make the model less impressive. It is a big problem in practical use though.

I am curious on what you mean by adversarial examples/triggers for humans in the domain of natural language. Off the top of my head, I can think of: * garden path sentences * highly recursive sentences Could you or anyone provide some other classes? The two classes above however can generally be understood by a large number of educated native speakers with time to think carefully. Humans also do not get derailed so b…

I don't think universal triggers exist, since at that point they are just language features. But there are plenty of less universal triggers

Let's imagine that that in the brain everything goes through a series of models, first tokenization into words, then we build something like an abstract syntax tree, then we analyse meaning in the context etc; and each time one of these steps reaches a nonsensical result we start over with additional parsing time allocated. It's probably not true, but close enough to be a useful model.

Now what you consider an adversarial example depends on how far down the stack it has to go until it's caught:

- "The old man the boat." fails in the early parsing steps. We reliably miscategorize old as adjective when it's a noun.

- "More people have been to Russia than I have, said Escher" goes a step further, it parses just fine but makes no sense. The tricky thing is that you might initially not notice that it makes no sense. This is about the level where AI is today.

- "Time flies like an arrow; fruit flies like a banana" makes perfect sense, but you could notice that the straight forward way to parse it leads to a non-sequitur and parsing it as "time-flies love eating arrows; fruit-flies love eating bananas" is probably a better way to parse it.

Of course that's just the parsing steps. You can trick human "sentiment analysis" by swapping words without changing the meaning. Compare "this bag is made from fake leather" to "this bag is made from vegan leather". PR and marketing have made a science out of how to make bad things sound good. Similarly PR is great at finding adversarial examples for reading comprehension, where they say one thing that's nearly universally understood to mean something different (or to mean nothing at all; or where something that seems to mean nothing at all actually means something very siginicant).

Of course we assume all text to be targeted to humans; so if something is widely misunderstood by humans we blame the sender for writing such a bad message; when it's widely misunderstood by AI we blame the AI for being so bad at reading.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#238

Earlier quoted context omitted.

The second one means "the enemy successfully got several of our aircrafts". Specifically, definition 3a or 3b for the verb form here: https://www.merriam-webster.com/dictionary/land So potentially the enemy captured the aircraft (3a) or destroyed them (3b).

Aircraft typically get captured on the ground, or get forced to land by threat of being shot down. “Landed”, for me, would require the enemy to actively land the plane, just as “landing a fish” requires both the fisherman’s action and moving the fish from water to land. I also wouldn’t use “landed” for destroying an enemy plane (neither by shooting it down nor by destroying it on the ground) That, realistically, leav…

Yes -- if the sentence had been "grounded the aircraft", then the meaning is obvious. But even though "land" is a synonym for "ground" I don't think there's an equivalence of meaning here. I'm struggling to find a sense in which "landing and enemy aircraft" is a meaningful concept short of jumping out of one plane to land on another one, removing the pilot, and landing the plane, which is a bit much for the single word "landed" to carry.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#239

Earlier quoted context omitted.

I think this is really interesting, because "the enemy landed several of our aircraft(s)" is the sort of sentence I'd have hauled a student up for using as a teacher, because 1) it's a none standard, arguably incorrect usage they've used either because they're a none native speaker or because they're trying to be clever and failing, and 2) because the plural of aircraft is aircraft. Nevertheless the author of this se…

If you teach others English, please learn the difference between "none" and "non". You mean "non-standard" in all your examples here (if British) or perhaps "nonstandard" (if American).

Yup. They made the same mistake in "none native" (sic).

I'll admit that, as a non-native speaker, this fills me with glee.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#240

Earlier quoted context omitted.

My understanding is that a lot of these really high performance models that reach for every percentage-point possible require an absurd amount of hardware - specifically an absurd amount of GPU memory. For example I have what I consider a fairly "high end" rig for being a hobbyist individual, with 32GB of RAM, i7 8700k, 1080ti - there's 0 chance their model would fit on my system. So I mean maybe if you have a ton of…

Maybe I'm oversimplifying, but it seems to me that once you have the model trained, it should be possible to partition it somehow when inferencing, to fit smaller machines. At least for a proof of concept it should be possible.

I'm not aware of any "partioning" strategies per se (at least during inference), but it's now common practice to distill a larger model to a smaller one by either (a) training a smaller "student" network to replicate the larger "teacher" network, or (b) pruning smaller weights from the larger network to reduce the size.

Just brainstorming here, but a vanilla network partition strategy might be to load each layer's weight into memory and perform the forward pass sequentially. I think that would be prohibitively slow - some of these models (e.g. BERT) can already take up to 3-4 seconds to perform a single forward pass on a CPU, and that's with all model weights already loaded into main memory. I suspect fetching/loading each layer separately would blow this out by an order of magnitude.

Post reply on HN