Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

super.gluebenchmark.com

Post reply on HN
Print view View original

191–200 of 246 posts

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#191

Post by nmeofthestate »

TylerE wrote: post #51
LaMarseillaise wrote: post #46

Earlier quoted context omitted.

If taking the "captured" interpretation, I think it could be reasonably inferred that they successfully landed the aircraft at an airfield afterwards (same meaning). This was my initial read of it and it does not seem strange to me on reflection. I would like also to point out that even if we do interpret the second as meaning "destroyed", the first could then be interpreted as a combat aviator shooting down an oppos…

Landed in the sense of a fisherman landing a marlin.

So at the end of the process they were in possession of the enemy aircraft. Maybe they jumped across in mid-air and wrestled it off the other pilot.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#192

Post by Veedrac »

Al-Khwarizmi wrote: post #172

This surprised me a bit, on the creation of the corpus they use for training: "We removed any page that contained any word on the “List of Dirty, Naughty, Obscene or Otherwise Bad Words”." I don't understand this decision. This list contains words that can be used in a perfectly objective sense, like "anus", "bastard", "erotic", "eunuch", "fecal", etc. I can understand that they want to avoid websites full of expleti…

They say they removed pages, not websites. Having false positives isn't a problem when you're still left with 750GB of data—quality matters more than slightly higher quantity at that point.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#193

Post by z3t4 »

Huppie wrote: post #117
rladd wrote: post #96

Earlier quoted context omitted.

My mother got a perfect 800 score on the GRE English test many years ago when she wanted to go back to graduate school after her children were grown up enough (highschool/college age). She told me that the way she got her perfect score was by realizing when the questions were wrong and thinking of what answer the test creators believed to be correct. She had to outguess the test creators and answer the questions wron…

I've had the 'pleasure' of taking some 'Microsoft certifications' at various companies I worked at in the past and this sounds extremely familiar. "I probably won't ever do it like that and/or there's a syntax error in all four of the answers... but this is the answer you want to hear. It's wrong, mind you, but it's what you want to hear."

I'm a spatial thinker, and I got a similar problem, I see all answers as correct. Eg. which one follows this sequence, and I can find a pattern to all alternatives. And I have to figure out which option the test author think is correct.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#194

Post by Al-Khwarizmi »

Veedrac wrote: post #192
Al-Khwarizmi wrote: post #172

This surprised me a bit, on the creation of the corpus they use for training: "We removed any page that contained any word on the “List of Dirty, Naughty, Obscene or Otherwise Bad Words”." I don't understand this decision. This list contains words that can be used in a perfectly objective sense, like "anus", "bastard", "erotic", "eunuch", "fecal", etc. I can understand that they want to avoid websites full of expleti…

They say they removed pages, not websites. Having false positives isn't a problem when you're still left with 750GB of data—quality matters more than slightly higher quantity at that point.

Sorry, I was thinking about pages even though I said websites. Native language interference (typically, we use the same term for pages and websites in my language).

Anyway, my point is not a matter of quantity. The way they're doing it, they have 750 GB of data, but they have exactly zero data that talks about bastards, fecal transplants, etc. So they may have a hard time answering questions about those specific subjects.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#195

Post by Huppie »

Double_a_92 wrote: post #189
Huppie wrote: post #117

Earlier quoted context omitted.

I've had the 'pleasure' of taking some 'Microsoft certifications' at various companies I worked at in the past and this sounds extremely familiar. "I probably won't ever do it like that and/or there's a syntax error in all four of the answers... but this is the answer you want to hear. It's wrong, mind you, but it's what you want to hear."

Sometimes the questions are also just broken. I.e. asking you to select the things that do not apply, but the answer would have been to opposite.

Back when I took the 'C# certification' (70-483 I think?) there were multiple questions in the style 'which of the following answers will make the program compile', where all four answers had a syntax error, or the program had a syntax error at a different line that would cause an issue regardless of your answer.

I tried the dispute process but it's basically impossible to dispute / report broken questions unless you have a photographic memory.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#196

Post by SiVal »

hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

Yes, the second should be, "The enemy downed several of our aircraft." Landed can be used to mean "bagged," as in, "We finally landed the Smith account," (it's a fishing term), but it should not be used in this figurative sense when referring to aircraft, because of the obvious confusion with the common, concrete sense of the word. And, yes, it should be aircraft.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#197

Post by otakucode »

ekianjo wrote: post #120
andrewstuart2 wrote: post #64

Earlier quoted context omitted.

One of my favorite examples that I heard in a David Rock talk which I can no longer find on youtube: "Time flies like an arrow": Time moves swiftly and in one direction. Record the speed of flies in the same way you would an arrow. Time flies, which are a kind of fly, are fond of an arrow. (e.g. Time flies like an arrow, fruit flies like a banana).

A good demonstration that context (and cultural conditioning) is everything to understand what a text actually means.

Not to mention something that almost all NLP systems are resounding terrible at - short-term memory. If we've been talking about corporate financials for an hour and I say 'Justify the margins', it should be crystal clear what I mean. But most automated systems try to operate without a hint of memory or 'state' being tracked.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#199

Post by m3at »

hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

I think people are digging too deep for an answer here... it seem to me to be a simple mistake, which on the scale at which they're evaluating those models is not statistically significant.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#200

Post by Frondo »

archontes wrote: post #53
bobbyi_settv wrote: post #12

Earlier quoted context omitted.

I've never seen "landed" used as in the second sentence, but I was definitely able to understand from context that it was not being used to mean the same thing as in the first sentence.

You've never landed a fish?

I haven't, though I'm familiar with that use of "landed" for fish.

As a lifelong native speaker (PNW English), I've also never heard "landed" used to refer to shooting down or capturing enemy airplanes. I could understand it from context, which is what I suppose the software is also going for, but I'd mark it with a red pen if someone showed me that sentence, just for clarity's sake (i.e. understandable from context but should be replaced).

Post reply on HN