Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline
super.gluebenchmark.com
Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline
1–10 of 246 posts
Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline
#2Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline
#3"The General Language Understanding Evaluation (GLUE) benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems."
"We take into account the lessons learnt from original GLUE benchmark and present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, improved resources, and a new public leaderboard."
Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline
#4There was an article[1] posted to HN recently about these benchmarks, and it was pretty skeptical.
Regarding SuperGLUE specifically, it asked:
"Indeed, Bowman and his collaborators recently introduced a test called SuperGLUE that's specifically designed to be hard for BERT-based systems. So far, no neural network can beat human performance on it. But even if (or when) it happens, does it mean that machines can really understand language any better than before? Or does just it mean that science has gotten better at teaching machines to the test?"
[1] - https://www.quantamagazine.org/machines-beat-humans-on-a-rea...
Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline
#5I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks.
One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/)
One of their examples, though, didn't make any sense to me:
1. The pilot managed to land the airplane safely
2. The enemy landed several of our aircrafts
It says that the word "land" does NOT mean the same thing in those sentences. I am a native English speaker, and I honestly don't understand what they are thinking the second sentence means. Shot them down? If so, I have never heard "landed" used in that context, and it appears neither has Merriam-Webster. Also, the plural of aircraft is just "aircraft", without the s.
Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline
#6Possibly dumb question: How do you ensure there's no data leakage when benchmarking transfer learning techniques? Is that even a problem anymore when the whole point is to learn "common sense" knowledge?
For example their “Colossal Clean Crawled Corpus” (C4), a dataset consisting of hundreds of gigabytes of clean English text scraped from the web, might contain much of the same information as the benchmark datasets, which I presume is also scraped from the web.
Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline
#7I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…
I think it's landed in the same sense as "landed a deal": got, or achieved, in this case achieving shooting them down.
For me, my first read of the sentence would definitely be that it means shot down.
Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline
#8I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…
I think they might going for 'landed' as in 'landed a deal'. Maybe?
Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline
#9I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…
In option 1, the aircraft met the ground gently and safely.
In option 2, the aircraft met the ground violently and lethally.
Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline
#10I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…
The only possible explanation I can think of is this.
3. a : to catch and bring in
// land a fish
b : gain, secure
// land a job landed the leading role
imagine enemy soldiers capturing a base or hangar ship including the aircraft.
It's kind of a stretch though.