Solving Verbal Questions in IQ Test by Knowledge-Powered Word Embedding [pdf] • Hacker News Board

Re: Solving Verbal Questions in IQ Test by Knowledge-Powered Word Embedding [pdf]

#11

Post by ganz » Sat, Jun 13, 2015, 5:32 AM UTC

dschiptsov wrote: post #9

joe_the_user wrote: post #5

Earlier quoted context omitted.

I'm reading the article and it's talking about solving analogies, antonyms and similar problems ... as found on a standard IQ test. I understand this isn't intelligence and the title doesn't imply it but I'm not sure what your reference to decoding is about.

But it still seems like pattern recognition based on a training set only, which is, in my opinion, a task on a lever prior to intelligence, like what visual cortex does. It cannot make new reference between words to produce (infer) new antonyms and analogies (not presented in a training g set), which is intelligence.

The system isn't training on antonyms and analogies - it's training on wikipedia. It's learning the meaning (and multiple senses) for every word it can find.

The test they use to see if it actually learned what these words meant, in a limited sense, is to test it against a subset of verbal IQ tests (not what it was trained on!). You could ask it the antonym, synonym, or analogy for anything in English. This is an extension of word2vec / word embeddings.

That it beats the scores of college graduates impresses me.

Re: Solving Verbal Questions in IQ Test by Knowledge-Powered Word Embedding [pdf]

#12

Post by vonsydov » Sat, Jun 13, 2015, 5:44 AM UTC

This is not a verbal comprehension test.

Re: Solving Verbal Questions in IQ Test by Knowledge-Powered Word Embedding [pdf]

#13

Post by Qantourisc » Sat, Jun 13, 2015, 6:16 AM UTC

dschiptsov wrote: post #9

joe_the_user wrote: post #5

Earlier quoted context omitted.

I'm reading the article and it's talking about solving analogies, antonyms and similar problems ... as found on a standard IQ test. I understand this isn't intelligence and the title doesn't imply it but I'm not sure what your reference to decoding is about.

But it still seems like pattern recognition based on a training set only, which is, in my opinion, a task on a lever prior to intelligence, like what visual cortex does. It cannot make new reference between words to produce (infer) new antonyms and analogies (not presented in a training g set), which is intelligence.

Personally I'm not sure this helps for generic AI (directly). But could serve as an autonomies piece to help the generic AI with linquesitics (especially on shaving CPU cycles)? A bit like you can use your visual memory for certain things?

Re: Solving Verbal Questions in IQ Test by Knowledge-Powered Word Embedding [pdf]

#14

Post by jyzzmoe » Sat, Jun 13, 2015, 6:28 AM UTC

JuliaLang wrote: post #8

jyzzmoe wrote: post #2

And yet ... http://arxiv.org/pdf/1412.1897v1.pdf%EF%BB%BF

I can break humans too: https://en.wikipedia.org/wiki/List_of_optical_illusions

Humans are able to recognize these as optical illusions. (otherwise we wouldn't be calling them "illusions") Further, none of the optical illusions shown are so wildly "off" and "crazy" as the algorithmic goofs demonstrated in the paper referenced above.

Re: Solving Verbal Questions in IQ Test by Knowledge-Powered Word Embedding [pdf]

#15

Post by joe_the_user » Sat, Jun 13, 2015, 6:32 AM UTC

dschiptsov wrote: post #9

joe_the_user wrote: post #5

Earlier quoted context omitted.

I'm reading the article and it's talking about solving analogies, antonyms and similar problems ... as found on a standard IQ test. I understand this isn't intelligence and the title doesn't imply it but I'm not sure what your reference to decoding is about.

But it still seems like pattern recognition based on a training set only, which is, in my opinion, a task on a lever prior to intelligence, like what visual cortex does. It cannot make new reference between words to produce (infer) new antonyms and analogies (not presented in a training g set), which is intelligence.

Well, it is teaching a computer to do well on an "intelligence test" and ironically many of the ways that humans use to distinguish each other's intelligence tend to not measure the unique, flexible and adaptable properties of human intelligence and rather tend to be tests of more computer-like behavior in human - playing chess well was for a long time considered a measure of high intelligence, for example.

Re: Solving Verbal Questions in IQ Test by Knowledge-Powered Word Embedding [pdf]

#16

Post by [deleted] » Sat, Jun 13, 2015, 6:33 AM UTC

clickok wrote: post #7

Interesting and cool. It's almost like a magic trick, particularly in the sense that an almost unbelievable result has been achieved through putting in a serious amount of effort. It's not just a gimmick, though-- projects like this demonstrate the steady creep of progress on seemingly unsolvable problems[1]. But of course, there's thousands of such problems out there, with millions of potential customers ready to ma…

[deleted]

Re: Solving Verbal Questions in IQ Test by Knowledge-Powered Word Embedding [pdf]

#17

Post by DavidSJ » Sat, Jun 13, 2015, 6:44 AM UTC

jyzzmoe wrote: post #14

JuliaLang wrote: post #8

Earlier quoted context omitted.

I can break humans too: https://en.wikipedia.org/wiki/List_of_optical_illusions

Humans are able to recognize these as optical illusions. (otherwise we wouldn't be calling them "illusions") Further, none of the optical illusions shown are so wildly "off" and "crazy" as the algorithmic goofs demonstrated in the paper referenced above.

They're obviously crazy to us. Our illusions might look obviously crazy to them.

And generally humans don't natively know when they're experiencing an optical illusion. They have to be taught it. And in either case, it's not the human vision system that learns the lesson, it's some other part of the brain that learns to discount the vision system's conclusions.

Re: Solving Verbal Questions in IQ Test by Knowledge-Powered Word Embedding [pdf]

#18

Post by breuderink » Sat, Jun 13, 2015, 7:56 AM UTC

JuliaLang wrote: post #8

jyzzmoe wrote: post #2

And yet ... http://arxiv.org/pdf/1412.1897v1.pdf%EF%BB%BF

I can break humans too: https://en.wikipedia.org/wiki/List_of_optical_illusions

Further, these inputs were /optimized/ to confuse the particular classification software. If only we would be able to optimise optical illusions for a given human viewer...

Re: Solving Verbal Questions in IQ Test by Knowledge-Powered Word Embedding [pdf]

#19

Post by empiricus » Sat, Jun 13, 2015, 8:02 AM UTC

Houshalter wrote: post #4

jyzzmoe wrote: post #2

And yet ... http://arxiv.org/pdf/1412.1897v1.pdf%EF%BB%BF

Not really relevant to this. That deals with images, where you can perturb a huge number of pixels slightly to exploit weird edge cases. Even linear classifiers break on it, e.g. https://karpathy.github.io/2015/03/30/breaking-convnets/

I'm pretty sure that if you move just a little the image, the convnet is no longer fooled. So this kind of exploit works only for completely stationary images, and it is very to overcome.

Re: Solving Verbal Questions in IQ Test by Knowledge-Powered Word Embedding [pdf]

#20

Post by compbio » Sat, Jun 13, 2015, 8:10 AM UTC

huac wrote: post #6

How reliable is MTurk for an IQ test? I presume the respondents don't put in that much effort on these quizzes.

MTurk is representative of the human population. I hear it often that people presume that MTurkers do not put in any effort for a few cents. But for most it is a hobby/nice pastime. They do it to make a few bucks, instead of playing chess or reading a book. Most people do not get a nickel playing chess, though really put in the effort. MTurkers are similar.

See here for an MIT experiment in blurry text transcription: http://groups.csail.mit.edu/uid/deneme/?p=329 for the unexpected accuracy resulting from crowd-sourcing.