AI’s Language Problem • Hacker News Board

Re: AI’s Language Problem

#191

Post by jonkho » Wed, Aug 10, 2016, 12:52 AM UTC

erikpukinskis wrote: post #84

No one would ever imagine that locking a baby in a featureless room with a giant stack of books would give them general intelligence. I don't understand why AI researchers think it will work for AIs. They need bodies that are biologically connected with the rest of the biosphere, with an intrinsic biological imperative, if they are ever to understand the world. I'm not saying they have to be exactly like us, but they…

I second this. To further illustrate that semantics is far more than syntax and that emotions are an inseparable part of understanding language, do the following experiment: take any word or concept and look up its definition in a dictionary. Take note of the words used in the definition and look up the meaning of those words. Repeat recursively and if you go far enough down the tree you will find that all words lead to self-recursive definitions that state "it is a feeling of" this or that. This is proof that semantics of language - and intelligence required to grasp it's semantics - is inseparable from subjective states.

Re: AI’s Language Problem

#192

Post by eli_gottlieb » Wed, Aug 10, 2016, 1:17 AM UTC

Houshalter wrote: post #102

erikpukinskis wrote: post #84

No one would ever imagine that locking a baby in a featureless room with a giant stack of books would give them general intelligence. I don't understand why AI researchers think it will work for AIs. They need bodies that are biologically connected with the rest of the biosphere, with an intrinsic biological imperative, if they are ever to understand the world. I'm not saying they have to be exactly like us, but they…

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vecto…

Learning the statistics of language is not going to tell the ML model anything about the underlying stuff to which the language actually refers. It will need actual "sense-data" to do that. For instance, to get a model that generates image captions, you need to train it with actual images.

If the much-vaunted "general intelligence" consists in both vague and precise causal reasoning and optimal control with respect to objects in the real world, then no, it obviously cannot be done with mere language. At least one sensor and one effector will be needed to train an ML/AI model to perform those tasks.

Re: AI’s Language Problem

#193

Post by Houshalter » Wed, Aug 10, 2016, 1:18 AM UTC

jonkho wrote: post #190

erikpukinskis wrote: post #84

No one would ever imagine that locking a baby in a featureless room with a giant stack of books would give them general intelligence. I don't understand why AI researchers think it will work for AIs. They need bodies that are biologically connected with the rest of the biosphere, with an intrinsic biological imperative, if they are ever to understand the world. I'm not saying they have to be exactly like us, but they…

I second this. To further illustrate that semantics is far more than syntax and that emotions are an inseparable part of understanding language, do the following experiment: take any word or concept and look up its definition in a dictionary. Take note of the words used in the definition and look up the meaning of those words. Repeat recursively and if you go far enough down the tree you will find that all words lead…

What does the experiment prove? Of course dictionaries are circular. But that doesn't mean they don't contain huge amounts of information about the world, and about language. Information an AI could infer without any interaction with the outside world at all.

Re: AI’s Language Problem

#194

Post by eli_gottlieb » Wed, Aug 10, 2016, 1:24 AM UTC

dmreedy wrote: post #134

Houshalter wrote: post #133

Earlier quoted context omitted.

>There is a class of functions that are simply not learnable without some prerequisite knowledge. This is directly analogous to a one-time pad in crypto. It is entirely possible that the function 'language' is in this class of unlearnable functions. I don't understand how this could possibly be the case. We can already make great progress on language understanding with simple methods like word2vec, or perhaps even ma…

It can be the case if Chomsky was right, and Universal Grammar and other similar structures are a thing. That would mean that part of our ability to understand language comes from the particular structure of our brain (which everyone seems to by and large share). That would mean that some of our ability to understand language is genetic in nature, by whatever means genes direct the structure of brain development.

Nah, it's mostly just hierarchical probability models.

http://www.ncbi.nlm.nih.gov/pubmed/24977647

Re: AI’s Language Problem

#195

Post by Houshalter » Wed, Aug 10, 2016, 1:25 AM UTC

eli_gottlieb wrote: post #192

Houshalter wrote: post #102

Earlier quoted context omitted.

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vecto…

Learning the statistics of language is not going to tell the ML model anything about the underlying stuff to which the language actually refers . It will need actual "sense-data" to do that. For instance, to get a model that generates image captions, you need to train it with actual images. If the much-vaunted "general intelligence" consists in both vague and precise causal reasoning and optimal control with respect…

There is nothing magical about "sense data". A video is just a bunch of 1's and 0's, just like text data. A model of video data is not superior in any way to one of text data, they are just different.

The internet is so large and so comprehensive (especially if you include digitized books and papers, e.g. libgen or google books) that I doubt any important information that can be learned through video data, can't be obtained through text data.

Re: AI’s Language Problem

#196

Post by rspeer » Wed, Aug 10, 2016, 1:27 AM UTC

Houshalter wrote: post #129

kafkaesq wrote: post #120

Earlier quoted context omitted.

It's learning the meaning of words, and the relationships between them. Word2vec is definitely an impressive algorithm. But at the end of the day, it's just a tool that cranks out a fine-grained clustering of words based on (a proxy measure for) contextual similarity (or rather: an embedding in a high-dimensional space, which implicitly allows the words to be more easily clustered). And yes, some additive relations b…

Word2vec may be crude, but it demonstrates that you can learn non-trivial relationships between words with even such a simple algorithm. What is the meaning of a word, if not the relationship it has to other words? Gender was just an example. There are lots of semantic information learned by word2vec, and the vectors have shown to be useful in text classification and other uses. It can learn subtle stuff, like the re…

I use word2vec a lot, and things like it, and I've always found it overstated to say that it "learns the relationships between things".

You say, as many people do, that the operation "king - man + woman = queen" indicates that it understands that the relation between "king" and "queen" is the relation between "man" and "woman". But there is something much simpler going on.

What you're asking it for, in particular, is to find the vector represented by "king - man + woman", and find the vector Q with the highest dot product with to this synthetic vector, out of a restricted vocabulary.

The dot product is distributive, so distribute it: you want to find the maximum value of (king * Q) - (man * Q) + (woman * Q).

So you want to find a vector that is like "king" and "woman", and not like "man", and is part of the extremely limited vocabulary that you use for the traditional word2vec analogy evaluation, but not one of the three words you used in the question. (All of these constraints are relevant.) Big surprise, the word that fits the bill is "queen".

(I am not the first to do this analysis, but I've heard it from enough people that I don't know who to credit.)

It's cool that you can use sums of similarities between words to get the right answer to some selected analogy problems. Really, it is. It's a great thing to show to people who wouldn't otherwise understand why we care so much about similarities between words.

But this is not the same thing as "solving analogies" or "understanding relationships". It's a trick where you make a system so good at recognizing similarities that it doesn't have to solve analogies or understand relationships to solve the very easy analogy questions we give it.

Re: AI’s Language Problem

#197

Post by rspeer » Wed, Aug 10, 2016, 1:32 AM UTC

Cybiote wrote: post #159

Houshalter wrote: post #129

Earlier quoted context omitted.

Word2vec may be crude, but it demonstrates that you can learn non-trivial relationships between words with even such a simple algorithm. What is the meaning of a word, if not the relationship it has to other words? Gender was just an example. There are lots of semantic information learned by word2vec, and the vectors have shown to be useful in text classification and other uses. It can learn subtle stuff, like the re…

But it and methods like it are still very limited in what they can learn. For example, they can't learn relations involving antonyms. They can't tell apart hot from cold or big from small.

In fact, it only learns relationships that are a linear combination of "similar to X" and "not similar to Y".

Re: AI’s Language Problem

#198

Post by djokkataja » Wed, Aug 10, 2016, 1:35 AM UTC

mannykannot wrote: post #183

erikpukinskis wrote: post #84

No one would ever imagine that locking a baby in a featureless room with a giant stack of books would give them general intelligence. I don't understand why AI researchers think it will work for AIs. They need bodies that are biologically connected with the rest of the biosphere, with an intrinsic biological imperative, if they are ever to understand the world. I'm not saying they have to be exactly like us, but they…

Helen Keller's life argues against the proposition that a machine requires the same sort of interaction with its environment that the average human experiences, before it can achieve intelligence.

She was blind and deaf, but she still had enormous amounts of tactile information and the ability to physically interact with her environment. Moreover, those sensory elements were critical for her to finally be able to start learning language--she finally caught on that the signs another person was making in one hand represented the water being run over her other hand. And she didn't make that breakthrough until she was 7 years old, and only then began to learn with the persistent help of an instructor.[0]

I would say that Helen Keller's life argues that a intelligent machine must be able to have experiences and the capacity to associate experiences with language. The machine probably doesn't need all of the perceptual modalities that we have, as Helen Keller demonstrated, but it should probably have some similarities with our own so that there would be common ground for initiating communication about experiences. A machine with just a text in / text out interface has nothing in common with us.

[0] https://en.wikipedia.org/wiki/Helen_Keller#Early_childhood_a...

Re: AI’s Language Problem

#199

Post by mchaver » Wed, Aug 10, 2016, 1:37 AM UTC

grondilu wrote: post #5

It seems to me that a full mastery of language requires a grasp of semantics, that is the ability to understand what a sentence means. I doubt it's possible to do that without having basic common sense along with an overall representation of the world, and that looks very close to strong AI, imho. So I'm not surprised computers keep on struggling with language applications. Once they succeed strong AI will not be muc…

To grasp semantics of human language I would think that AI would have to an understanding of the world from a human experience. So we would need to simulate human experience for an AI. Anyone know of any work on this?

Re: AI’s Language Problem

#200

Post by Houshalter » Wed, Aug 10, 2016, 1:44 AM UTC

DonaldFisk wrote: post #180

Houshalter wrote: post #133

Earlier quoted context omitted.

>There is a class of functions that are simply not learnable without some prerequisite knowledge. This is directly analogous to a one-time pad in crypto. It is entirely possible that the function 'language' is in this class of unlearnable functions. I don't understand how this could possibly be the case. We can already make great progress on language understanding with simple methods like word2vec, or perhaps even ma…

Wouldn't you still need to attach meanings to the words though? How could an AI system ever understand, for example, the Voynich Manuscript? There's plenty of text in it, and encryption methods when it was written weren't particularly strong. Or how would a person do if they were locked in a room with lots of books written in a language unknown to them?

Infact someone has run word2vec on the Voynich manuscript: http://blog.christianperone.com/2016/01/voynich-manuscript-w... (web archive while it's down: http://web.archive.org/web/20160205003817/http://blog.christ...) Such methods could someday completely decode the thing, but for now they just show the relationships between words and different clusters of words, not their meaning.

Of course we have no idea how the Voynich manuscript is encrypted (which would make the assumptions of word2vec wrong), or if it even has any meaning at all. And it's an incredibly small dataset compared to modern text corpuses, so there is probably significant uncertainty and overfitting. And other problems like inconsistent spellings, many errors in transcriptions, etc. But in principle this is a good strategy.

>how would a person do if they were locked in a room with lots of books written in a language unknown to them?

If you spent all day reading them, for years, and you somehow didn't get bored and kept at it, eventually you would start to see the patterns. You would learn how "slithy toves" are related to "brillig", even if you have no idea how that would translate to English. Study it long enough, and you may even be able to produce text in that language, indistinguishable from the real text. You may be able to predict the next word in a sentence, and identify mistakes, etc. Perhaps carry out a conversation in that language.

And I think eventually you would understand what the words mean, by comparing the patterns to those found in English. Once you have guesses for translations of just a few words, you can translate the rest. Because you know the relationships between words, and so knowing one word constrains the possibilities of what the other words can be.

If the translation it produces is nonsense, the words you guessed must have been wrong, and you can try again with other words. Eventually you will find a translation that isn't nonsense, and there you go. This would be very difficult for humans, because the number of hypotheses to test is so large, and analyzing text takes forever. Computers can do it at lightspeed though.