Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

super.gluebenchmark.com

Post reply on HN
Print view View original

151–160 of 246 posts

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#151

Post by BiasRegularizer »

ilaksh wrote: post #95

The AIs in the benchmark are all trained exclusively on text, correct? My assumption has always been that to get human-level understanding, the AI systems need to be trained on things like visual data in addition to text. This is because there is a fair amount of information that is not encoded at all in text, or at least is not described in enough detail. I mean, humans can't learn to understand language properly wi…

I think maybe CLEVR[0] dataset is what you are talking about?

Keep in mind that a most of the current ML systems have diverged from biology. A majority of the recent breakthroughs come from mathematics, the rational is that just because human brain does it in a certain way does not necessarily mean it is the only way to do it.

[0] https://cs.stanford.edu/people/jcjohns/clevr/

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#152

Post by sjg007 »

_II__II_ wrote: post #57
hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

The example directly below that: "Justify the margins" and "The end justifies the means" is the one I find dubious. Obviously the former could mean to format a document, but those exact words in that structure could be a demand for someone to justify a financial margin for example. It is both true and false depending on the context.

The ends justify the means.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#153

Post by scarletmantis »

hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

I understood 'landed' as an euphemism for 'shot down'.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#154

Post by alex_young »

hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

The plural of aircraft is aircraft. Not aircrafts. https://www.grammar-monster.com/plurals/plural_of_aircraft.h...

Maybe the _examples_ for a language test should be grammatically correct?

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#155

Post by BiasRegularizer »

alex_young wrote: post #154
hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

The plural of aircraft is aircraft. Not aircrafts. https://www.grammar-monster.com/plurals/plural_of_aircraft.h... Maybe the _examples_ for a language test should be grammatically correct?

True, but having some noise in the label is actually good for generalization. If it's only learned on perfectly correct sentences then its tolerance for mistakes will be very low.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#156

Post by kees99 »

Doxin wrote: post #144
QuinnWilton wrote: post #90

Earlier quoted context omitted.

It sounds like you're talking about garden-path sentences [0], and in particular: "time flies like an arrow; fruit flies like a banana" [1]. These are sentences whose structure tricks the reader into making an incorrect parse. My favourite of these has always been: "The horse raced past the barn fell". [0] https://en.wikipedia.org/wiki/Garden-path_sentence [1] https://en.wikipedia.org/wiki/Time_flies_like_an_arrow;_f…

You know, I only just now got the second interpretation of that sentence. I always thought of it like "Time flies like an arrow (straight and in one direction), Fruit flies like a banana (when thrown)" Obvious in hindsight...

Same here, except it's comparing fly's flight trajectory to that of a banana is new to me.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#157

Post by IshKebab »

6gvONxR4sf7o wrote: post #58

One thing to always point out in these cases is that the human baseline isn't "how well people do at this task," like it's often hyped to be. It's "how well does a person quickly and repetitively doing this do, on average." The 'quickly and repetitively' part is important because we all make more boneheaded errors in this scenario. The 'on average' part is important because the errors the algo makes aren't just fewer…

Regarding the type of errors, it seems like the benchmark should be able to take that into account. That is, get a load of humans to do the task on the same specific examples, then for each example you know how hard it is, and what acceptable answers are (I bet a lot of the ground truth is wrong or ambiguous).

Then you can benchmark your AI but penalise it more heavily for getting things wrong that are obvious to a human.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#158

Post by nightnight »

This clearly demonstrates once again that Google is miles ahead of the competition in AI. I mean, they just have the best data.

If you want to have an every day example of Google's AI skills: Switch you phone's keyboard to GBoard, especially all iOS users, and you will face a night and day difference to any other keyboard esepcially the stock one. When using multiple languages at the same time the leap to other keyboards gets even bigger.

GBoard is my phone's killer app and if Google dropped it for iOS I'd left the same day to Android.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#159

Post by conjectures »

rladd wrote: post #96
hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

My mother got a perfect 800 score on the GRE English test many years ago when she wanted to go back to graduate school after her children were grown up enough (highschool/college age). She told me that the way she got her perfect score was by realizing when the questions were wrong and thinking of what answer the test creators believed to be correct. She had to outguess the test creators and answer the questions wron…

This does seem like the meta solution to most tests, particularly standardised tests :)

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#160

Post by wisty »

topspin wrote: post #127
eindiran wrote: post #14

Earlier quoted context omitted.

The second one means "the enemy successfully got several of our aircrafts". Specifically, definition 3a or 3b for the verb form here: https://www.merriam-webster.com/dictionary/land So potentially the enemy captured the aircraft (3a) or destroyed them (3b).

Would a native English speaker use the word "landed" in this way? In the context of aircraft? "Landed" is badly ambiguous here and several distinct meanings are plausible. Captured is the most natural word given your interpretation. Honestly that sentence -- the use of landed and that awful plural -- approaches engrish. Is that deliberate or is the use of English here just badly flawed? I can't see any other possibil…

Possibly, you could say the planes were landed, as in forced to stay on the ground (because of damage, fear of enemy fire, or damage to the runway). But grounded would be better.
Post reply on HN