Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

super.gluebenchmark.com

Post reply on HN
Print view View original

121–130 of 246 posts

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#121

Post by parksy »

LaMarseillaise wrote: post #46
eindiran wrote: post #14

Earlier quoted context omitted.

The second one means "the enemy successfully got several of our aircrafts". Specifically, definition 3a or 3b for the verb form here: https://www.merriam-webster.com/dictionary/land So potentially the enemy captured the aircraft (3a) or destroyed them (3b).

If taking the "captured" interpretation, I think it could be reasonably inferred that they successfully landed the aircraft at an airfield afterwards (same meaning). This was my initial read of it and it does not seem strange to me on reflection. I would like also to point out that even if we do interpret the second as meaning "destroyed", the first could then be interpreted as a combat aviator shooting down an oppos…

The benchmark is useful primarily because it puts humans and computers on a level playing field. Human readers will misinterpret written language, and human writers will poorly represent concepts.

The propensity to make mistakes in comprehension is unavoidable, humans only approach 90% accuracy, and computers are getting close to the same level of accuracy on the same base materials as humans.

The other way of testing would be to devise a test where there is only a single interpretation, where the context is clear, and there is no ambiguity in meaning. In that case a competent human and computer algorithm could be expected to answer all questions perfectly.

The purpose of this benchmark on the other hand is to test comprehension when meaning is not explicit and context clues are implied, something humans have had the advantage at over computers until quite recently. The computer won't be 100% accurate, but that's not the purpose of this test.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#125

Post by [deleted] »

rladd wrote: post #96
hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

My mother got a perfect 800 score on the GRE English test many years ago when she wanted to go back to graduate school after her children were grown up enough (highschool/college age). She told me that the way she got her perfect score was by realizing when the questions were wrong and thinking of what answer the test creators believed to be correct. She had to outguess the test creators and answer the questions wron…

[deleted]

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#126

Post by nl »

changoplatanero wrote: post #118
nl wrote: post #84

Earlier quoted context omitted.

These aren't really alternate _parses_ though (in the sense that they don't give different parse trees). They do highlight the different possible meanings of "with" though. I think "I eat my rice with chicken" vs "I eat my rice with children" vs "I eat my rice with chopsticks" is the canonical example here. There's a whole field in NLP involved in showing what changes happen to entities mentioned in a sentence as a a…

Wouldnt those be different parse trees? Like the "with X" could either be attached either to the verb or the noun

Yes, possibly.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#127

Post by topspin »

eindiran wrote: post #14
hn_throwaway_99 wrote: post #5

I didn't know anything about SuperGLUE before (turns out it's a benchmark for language understanding tasks), so I clicked around their site where they show different examples of the tasks. One "word in context" task is to look at 2 different sentences that have a common word and decide if that word means the same thing in both sentences or different things (more details here: https://pilehvar.github.io/wic/ ) One of…

The second one means "the enemy successfully got several of our aircrafts". Specifically, definition 3a or 3b for the verb form here: https://www.merriam-webster.com/dictionary/land So potentially the enemy captured the aircraft (3a) or destroyed them (3b).

Would a native English speaker use the word "landed" in this way? In the context of aircraft? "Landed" is badly ambiguous here and several distinct meanings are plausible. Captured is the most natural word given your interpretation.

Honestly that sentence -- the use of landed and that awful plural -- approaches engrish. Is that deliberate or is the use of English here just badly flawed? I can't see any other possibilities.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#128

Post by lultimouomo »

changoplatanero wrote: post #118
nl wrote: post #84

Earlier quoted context omitted.

These aren't really alternate _parses_ though (in the sense that they don't give different parse trees). They do highlight the different possible meanings of "with" though. I think "I eat my rice with chicken" vs "I eat my rice with children" vs "I eat my rice with chopsticks" is the canonical example here. There's a whole field in NLP involved in showing what changes happen to entities mentioned in a sentence as a a…

Wouldnt those be different parse trees? Like the "with X" could either be attached either to the verb or the noun

I think it's more clear if you say "I usually eat X with Y", i.e. Y it's either the company, the tool or the condiment that you eat with (contrasted with "I'm eating my X", where X is a dish like "rice with chicken")

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#129

Post by visarga »

ilaksh wrote: post #95

The AIs in the benchmark are all trained exclusively on text, correct? My assumption has always been that to get human-level understanding, the AI systems need to be trained on things like visual data in addition to text. This is because there is a fair amount of information that is not encoded at all in text, or at least is not described in enough detail. I mean, humans can't learn to understand language properly wi…

It's not just grounding the language in vision, but the embodiment, first person perspective and ability to interact with the environment. Humans have had the benefit of slowly evolving in a complex environment which is too expensive to recreate for artificial agents. We can only create very limited sims vs the real world.

Re: Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline

#130

Post by sgt101 »

topspin wrote: post #127
eindiran wrote: post #14

Earlier quoted context omitted.

The second one means "the enemy successfully got several of our aircrafts". Specifically, definition 3a or 3b for the verb form here: https://www.merriam-webster.com/dictionary/land So potentially the enemy captured the aircraft (3a) or destroyed them (3b).

Would a native English speaker use the word "landed" in this way? In the context of aircraft? "Landed" is badly ambiguous here and several distinct meanings are plausible. Captured is the most natural word given your interpretation. Honestly that sentence -- the use of landed and that awful plural -- approaches engrish. Is that deliberate or is the use of English here just badly flawed? I can't see any other possibil…

I think it's archaic; in the past a fishing reference would have been more common and widely understood.
Post reply on HN