AI’s Language Problem • Hacker News Board

Re: AI’s Language Problem

#121

Post by melloclello » Tue, Aug 09, 2016, 7:24 PM UTC

Houshalter wrote: post #102

erikpukinskis wrote: post #84

No one would ever imagine that locking a baby in a featureless room with a giant stack of books would give them general intelligence. I don't understand why AI researchers think it will work for AIs. They need bodies that are biologically connected with the rest of the biosphere, with an intrinsic biological imperative, if they are ever to understand the world. I'm not saying they have to be exactly like us, but they…

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vecto…

I can't say anything about the feasibility of what you describe, but something about the idea of being a being composed of pure text, no ability to perceive or visualise the world around me except through the medium of text, utterly horrifies me

Re: AI’s Language Problem

#122

Post by Animats » Tue, Aug 09, 2016, 7:26 PM UTC

In terms of DNA difference, humans are very close to mammals that don't have much language capability but demonstrate some degree of intelligence. This suggests that language is not as fundamental as some researchers claim.

I once told Rod Brooks, back when he was proposing "Cog" (look it up), that he'd done a really good insect robot, and the next step should be a good robot mouse. He said "I don't want to go down in history as the guy who built the world's best robot mouse". "Cog" was a dud, and Brooks went back to insect level AI in the form of robot vacuum cleaners.

We need more machines which successfully operate autonomously in the real world. Then they may need to talk to humans and each other. That might work.

The big problem in AI isn't language, anyway. It's consequences. We don't have common sense for robots. There's little or no understanding of the consequences of planned actions. We need to get this figured out before we can let robots do much.

Re: AI’s Language Problem

#123

Post by erikpukinskis » Tue, Aug 09, 2016, 7:30 PM UTC

Houshalter wrote: post #102

erikpukinskis wrote: post #84

No one would ever imagine that locking a baby in a featureless room with a giant stack of books would give them general intelligence. I don't understand why AI researchers think it will work for AIs. They need bodies that are biologically connected with the rest of the biosphere, with an intrinsic biological imperative, if they are ever to understand the world. I'm not saying they have to be exactly like us, but they…

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vecto…

> All of the information of our world is contained in text.

That's a bit overstated. I think maybe you mean to say "all of the information you'd need to be an effective citizen of our world is contained in text"? Or something similar? I think even that is too strong a claim, but it's at least understandable.

As stated the assertion doesn't make any sense. There is more information in a glass of milk than could be stored on all of the computers on earth, and Heisenberg showed that it's impossible to even record all of the information about a single particle.

Certainly there is no textual information available about my grandfather's eyes, but that information is accessible in the world for those who'll look. You seem to be underestimating the quantity of information you absorbed as a baby, just by reacting robotically to the people and events around you and absorbing the relationships between percepts and feelings.

Re: AI’s Language Problem

#124

Post by jackpirate » Tue, Aug 09, 2016, 7:33 PM UTC

morgante wrote: post #111

posterboy wrote: post #104

Earlier quoted context omitted.

I don't find the Turing Test convincing either, because someone smart enough to build it should be smart enough to recognize it from its answers. And if that depends on the intelligence of the questioner, whose intelligence is tested then, really? IIRC the test is a binary classifier, but intelligence is a spectrum that's fuzzy and therefore inherently hard to define. IE, how low is someone willing to assume the lack…

> I don't find the Turing Test convincing either, because someone smart enough to build it should be smart enough to recognize it from its answers. Why do you assume that? The creators of AlphaGo certainly couldn't beat it.

The Turing Test equivalent in the game of Go is not beating the computer. Instead, it is: given a history of moves, can you determine whether or not a computer was playing? Determining whether or not AlphaGo was playing seems like a relatively easy task to me for the designers. Since they have access to the AlphaGo system, they can just calculate the probability that each move corresponds to one AlphaGo would make.

Re: AI’s Language Problem

#125

Post by morgante » Tue, Aug 09, 2016, 7:38 PM UTC

jackpirate wrote: post #124

morgante wrote: post #111

Earlier quoted context omitted.

> I don't find the Turing Test convincing either, because someone smart enough to build it should be smart enough to recognize it from its answers. Why do you assume that? The creators of AlphaGo certainly couldn't beat it.

The Turing Test equivalent in the game of Go is not beating the computer. Instead, it is: given a history of moves, can you determine whether or not a computer was playing? Determining whether or not AlphaGo was playing seems like a relatively easy task to me for the designers. Since they have access to the AlphaGo system, they can just calculate the probability that each move corresponds to one AlphaGo would make.

> Instead, it is: given a history of moves, can you determine whether or not a computer was playing?

No, it's really not. For the Turing Test, the AI is meant to be adversarial—it's objective is to convince you that it is human.

AlphaGo's objective isn't to "play like a human," it is to win. If they gave it an objective of playing like a human, I'm sure AlphaGo could play in a way that would be indistinguishable from a human.

> Since they have access to the AlphaGo system, they can just calculate the probability that each move corresponds to one AlphaGo would make.

Peeking at the system/data is cheating. Obviously the person who sets up a Turing test knows which player is AI.

Re: AI’s Language Problem

#126

Post by Houshalter » Tue, Aug 09, 2016, 7:43 PM UTC

erikpukinskis wrote: post #123

Houshalter wrote: post #102

Earlier quoted context omitted.

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vecto…

> All of the information of our world is contained in text. That's a bit overstated. I think maybe you mean to say "all of the information you'd need to be an effective citizen of our world is contained in text"? Or something similar? I think even that is too strong a claim, but it's at least understandable. As stated the assertion doesn't make any sense. There is more information in a glass of milk than could be sto…

Well of course that's what I meant. Any information an average person knows is contained in text, somewhere. That is all common sense knowledge. Everything from detailed descriptions of trees, to the color of the sky, to the shape of the human face, etc. But also much more, like all of our scientific knowledge and written history. Billions of things the average person doesn't know.

Re: AI’s Language Problem

#127

Post by hacker42 » Tue, Aug 09, 2016, 7:45 PM UTC

Houshalter wrote: post #102

erikpukinskis wrote: post #84

No one would ever imagine that locking a baby in a featureless room with a giant stack of books would give them general intelligence. I don't understand why AI researchers think it will work for AIs. They need bodies that are biologically connected with the rest of the biosphere, with an intrinsic biological imperative, if they are ever to understand the world. I'm not saying they have to be exactly like us, but they…

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vecto…

I agree that learning to reason about the world likely does not require experience with motor control and proprioception (i.e. literally how babies do it), though I do think that you either need at least some sort of tempo-spatial experience (e.g. visual). Tempo-spatial representations are just extremely hard to convey by text only. You might get the idea of closeness by saying 'close is when two words are close in a sequence of words' and ordering by saying 'this word comes after that word', but I think it would be very difficult to extrapolate that concept to more than one dimension, and dimensions that actually have not just an ordering but also a metric (just think about our inability to reason about just the fourth dimension). You need rich representations of our 3+1 dimensional world to be able to reason about it and text only gives you perhaps "0.5" dimensions (because it lacks a metric, i.e. it does not convey durations in terms of the ticks of the recurrent network). But I doubt, too, that interaction with the world is necessary, in fact I think it would be rather easy for an AI to simply write motor programs in a programming language given unrestricted and noiseless memory once it has learned to reason about tempo-spatial patterns from just observing them and identifying them with our language-coded shared concept space. It is not constrained to real-time performance of actions as humans are, therefore it can take the much easier way of programming any interaction with the world as needed, on the fly. Our shared concept space likely sufficiently conveys our general (common sense) knowledge how these patterns are known to interact and evolve over time once rudimentary tempo-spatial representations are in place.

Re: AI’s Language Problem

#128

Post by dwaltrip » Tue, Aug 09, 2016, 7:52 PM UTC

Practicality wrote: post #4

If we use animals as a reference, I would say that consciousness is more fundamental than language, so most likely we need that in place before we can get AI to be able to effectively understand language.

Complete agreement. How the heck could an entity (machine or biological) communicate meaningful ideas of any sort without a comprehensive understanding of the world upon which those ideas rest?

Language isn't some side feature. It's a complicated interface layer that lives on top of an enormously rich, dynamic internal model of the world one lives in.

The article barely touches on this fundamental aspect.

Re: AI’s Language Problem

#129

Post by Houshalter » Tue, Aug 09, 2016, 7:55 PM UTC

kafkaesq wrote: post #120

Houshalter wrote: post #102

Earlier quoted context omitted.

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vecto…

It's learning the meaning of words, and the relationships between them. Word2vec is definitely an impressive algorithm. But at the end of the day, it's just a tool that cranks out a fine-grained clustering of words based on (a proxy measure for) contextual similarity (or rather: an embedding in a high-dimensional space, which implicitly allows the words to be more easily clustered). And yes, some additive relations b…

Word2vec may be crude, but it demonstrates that you can learn non-trivial relationships between words with even such a simple algorithm. What is the meaning of a word, if not the relationship it has to other words?

Gender was just an example. There are lots of semantic information learned by word2vec, and the vectors have shown to be useful in text classification and other uses. It can learn subtle stuff, like the relationship between countries, celebrities, etc. All that information is contained in a few hundred dimensions, which is tiny compared to the neurons in the brain.

Re: AI’s Language Problem

#130

Post by dmreedy » Tue, Aug 09, 2016, 7:58 PM UTC

Houshalter wrote: post #102

erikpukinskis wrote: post #84

No one would ever imagine that locking a baby in a featureless room with a giant stack of books would give them general intelligence. I don't understand why AI researchers think it will work for AIs. They need bodies that are biologically connected with the rest of the biosphere, with an intrinsic biological imperative, if they are ever to understand the world. I'm not saying they have to be exactly like us, but they…

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vecto…

> All of the information of our world is contained in text

Even if this were a true statement, it's still the case that it might not be enough. There is a class of functions that are simply not learnable without some prerequisite knowledge. This is directly analogous to a one-time pad in crypto. It is entirely possible that the function 'language' is in this class of unlearnable functions. While it may be the case that certain varieties of intelligence are learnable tabula rasa from a powerful neural net, the surface form of human natural language (the part your recommending measuring) may simply not have enough information in it to decode the whole picture. It is entirely possible that you need to supply some of your own information as well to the picture, in a specific manner so as to act as a kind of decryption key. A record needs a record player, even if you can make similar sounds with cassettes and CDs.

And so, I'm willing to bet that you simply cannot, using raw, uninformed statistical techniques, predict what word a human would say next. You need to understand more of the underlying structure of humans first.

I will agree, however, that the success towards the Hutter Prize is a valuable demonstration of AI progress. Simply because I believe that maximal compression and the kind of intelligence I'm talking about are one and the same thing. You need to offload as much of the semantic weight of the corpus into the encryption algorithm as you can. That means building a very complex model of natural language. And if you accept the premise that this model is not simply learnable by observing the surface form, then that means building Strong AI