Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI’s Language Problem

technologyreview.com

Post reply on HN
Print view View original

131–140 of 244 posts

Re: AI’s Language Problem

#131

Post by mcphage »

Houshalter wrote: post #129
kafkaesq wrote: post #120

Earlier quoted context omitted.

It's learning the meaning of words, and the relationships between them. Word2vec is definitely an impressive algorithm. But at the end of the day, it's just a tool that cranks out a fine-grained clustering of words based on (a proxy measure for) contextual similarity (or rather: an embedding in a high-dimensional space, which implicitly allows the words to be more easily clustered). And yes, some additive relations b…

Word2vec may be crude, but it demonstrates that you can learn non-trivial relationships between words with even such a simple algorithm. What is the meaning of a word, if not the relationship it has to other words? Gender was just an example. There are lots of semantic information learned by word2vec, and the vectors have shown to be useful in text classification and other uses. It can learn subtle stuff, like the re…

> What is the meaning of a word, if not the relationship it has to other words?

There's also the relationship it has with the world.

Re: AI’s Language Problem

#132

Post by Houshalter »

mcphage wrote: post #131
Houshalter wrote: post #129

Earlier quoted context omitted.

Word2vec may be crude, but it demonstrates that you can learn non-trivial relationships between words with even such a simple algorithm. What is the meaning of a word, if not the relationship it has to other words? Gender was just an example. There are lots of semantic information learned by word2vec, and the vectors have shown to be useful in text classification and other uses. It can learn subtle stuff, like the re…

> What is the meaning of a word, if not the relationship it has to other words? There's also the relationship it has with the world.

Well in my example the AI doesn't have to interact with the world at all. To pass the Turing test simply requires imitating a human, predicting what words they would say. You only need to know the relationships between words.

Re: AI’s Language Problem

#133

Post by Houshalter »

dmreedy wrote: post #130
Houshalter wrote: post #102

Earlier quoted context omitted.

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vecto…

> All of the information of our world is contained in text Even if this were a true statement, it's still the case that it might not be enough. There is a class of functions that are simply not learnable without some prerequisite knowledge. This is directly analogous to a one-time pad in crypto. It is entirely possible that the function 'language' is in this class of unlearnable functions. While it may be the case th…

>There is a class of functions that are simply not learnable without some prerequisite knowledge. This is directly analogous to a one-time pad in crypto. It is entirely possible that the function 'language' is in this class of unlearnable functions.

I don't understand how this could possibly be the case. We can already make great progress on language understanding with simple methods like word2vec, or perhaps even markov chains. There are tons of statistical patterns in text that can be learned by computers.

Re: AI’s Language Problem

#134

Post by dmreedy »

Houshalter wrote: post #133
dmreedy wrote: post #130

Earlier quoted context omitted.

> All of the information of our world is contained in text Even if this were a true statement, it's still the case that it might not be enough. There is a class of functions that are simply not learnable without some prerequisite knowledge. This is directly analogous to a one-time pad in crypto. It is entirely possible that the function 'language' is in this class of unlearnable functions. While it may be the case th…

>There is a class of functions that are simply not learnable without some prerequisite knowledge. This is directly analogous to a one-time pad in crypto. It is entirely possible that the function 'language' is in this class of unlearnable functions. I don't understand how this could possibly be the case. We can already make great progress on language understanding with simple methods like word2vec, or perhaps even ma…

It can be the case if Chomsky was right, and Universal Grammar and other similar structures are a thing. That would mean that part of our ability to understand language comes from the particular structure of our brain (which everyone seems to by and large share). That would mean that some of our ability to understand language is genetic in nature, by whatever means genes direct the structure of brain development.

Re: AI’s Language Problem

#135

Post by dwaltrip »

Houshalter wrote: post #132
mcphage wrote: post #131

Earlier quoted context omitted.

> What is the meaning of a word, if not the relationship it has to other words? There's also the relationship it has with the world.

Well in my example the AI doesn't have to interact with the world at all. To pass the Turing test simply requires imitating a human, predicting what words they would say. You only need to know the relationships between words.

Only knowing the relationships between words alone would just be a poor proxy for knowing the meanings of the words, e.g. what real world concepts the words attempt to represent. You might be able to get pretty far with this technique, but I would bet a lot of money you would not be able to get reliable, in-depth human level communication. The system needs to have an understanding of the world.

And then there is the fundamentally dynamic aspect of language, which strengthens the need for a rich understanding of the world that words describe and convey.

Re: AI’s Language Problem

#136

Post by posterboy »

morgante wrote: post #111
posterboy wrote: post #104

Earlier quoted context omitted.

I don't find the Turing Test convincing either, because someone smart enough to build it should be smart enough to recognize it from its answers. And if that depends on the intelligence of the questioner, whose intelligence is tested then, really? IIRC the test is a binary classifier, but intelligence is a spectrum that's fuzzy and therefore inherently hard to define. IE, how low is someone willing to assume the lack…

> I don't find the Turing Test convincing either, because someone smart enough to build it should be smart enough to recognize it from its answers. Why do you assume that? The creators of AlphaGo certainly couldn't beat it.

Good question. Someone beat it. He and his games as training sets were part of the development of AlphaGo development.

I edited the post, did you read that? You are making my point, you can't bootstrap a definition for artificial intelligence by comparison to humans, when human intelligence is not well defined either.

Re: AI’s Language Problem

#137

Post by jacobrobbins »

Houshalter wrote: post #102
erikpukinskis wrote: post #84

No one would ever imagine that locking a baby in a featureless room with a giant stack of books would give them general intelligence. I don't understand why AI researchers think it will work for AIs. They need bodies that are biologically connected with the rest of the biosphere, with an intrinsic biological imperative, if they are ever to understand the world. I'm not saying they have to be exactly like us, but they…

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vecto…

> All of the information of our world is contained in text.

This statement is false. There is a well known thought experiment called Mary’s Room the gist of which is that knowing all conceivable scientific knowledge about how humans perceive color is still not a substitute for being a human and perceiving the color red: https://philosophynow.org/issues/99/What_Did_Mary_Know

The experience of seeing red is an example of what is called “qualia”.

In Google AI systems that identify cats, birds, etc it is reasonable to imagine AI technology evolving towards systems that can discuss those objects at the level of a typical person. However with an AI based on text only there is no possibility of that. It would be like discussing color with a blind person or sound with a deaf person.

Re: AI’s Language Problem

#138

Post by orthoganol »

Houshalter wrote: post #102
erikpukinskis wrote: post #84

No one would ever imagine that locking a baby in a featureless room with a giant stack of books would give them general intelligence. I don't understand why AI researchers think it will work for AIs. They need bodies that are biologically connected with the rest of the biosphere, with an intrinsic biological imperative, if they are ever to understand the world. I'm not saying they have to be exactly like us, but they…

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vecto…

I think if AGI were possible from the basic statistical NLP techniques outlined in most advanced NLP textbooks, it would have already happened a decade ago.

Re: AI’s Language Problem

#139

Post by sgnelson »

jacobrobbins wrote: post #137
Houshalter wrote: post #102

Earlier quoted context omitted.

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vecto…

> All of the information of our world is contained in text. This statement is false. There is a well known thought experiment called Mary’s Room the gist of which is that knowing all conceivable scientific knowledge about how humans perceive color is still not a substitute for being a human and perceiving the color red: https://philosophynow.org/issues/99/What_Did_Mary_Know The experience of seeing red is an example…

To further this comment, also see Tacit Knowledge, of which the very definition is essentially knowledge that cannot or is extremely difficult to transfer through words alone.

https://en.wikipedia.org/wiki/Tacit_knowledge

Re: AI’s Language Problem

#140

Post by milcron »

thaumasiotes wrote: post #64
ideonexus wrote: post #11

I think about the AI language problem a lot while raising my kids. The article notes the word "forever" and how an AI must distinguish the literal from the figurative meaning of the word in context. My five-year-old still doesn't grasp the literal meaning of this word as "never-ending." To him, "forever" is simply a very very long time. He has the same problem with the concept of "infinity," where the word means both…

> His young mind has not yet recognized the paradox that "infinity" is the biggest number, so what does it mean when I say, "Infinity plus one"? Paradox? In the extended reals, where infinity is the biggest number, infinity plus one gets you infinity, just as you'd expect. In, say, the study of ordinal numbers, where there are many infinite quantities, it doesn't make any sense to talk about "the biggest number".

A paradox is "a statement or proposition that seems self-contradictory or absurd but in reality expresses a possible truth". If you encounter a paradox, it tells you more about how your framework of thinking is flawed than it does about reality.

"Infinity plus one" is a paradox to a child who believes that infinity is a finite number. When they realize what infinity actually means, the paradox will be resolved.

Post reply on HN