Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI’s Language Problem

technologyreview.com

Post reply on HN
Print view View original

141–150 of 244 posts

Re: AI’s Language Problem

#141

Post by sgnelson »

kafkaesq wrote: post #120
Houshalter wrote: post #102

Earlier quoted context omitted.

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vecto…

It's learning the meaning of words, and the relationships between them. Word2vec is definitely an impressive algorithm. But at the end of the day, it's just a tool that cranks out a fine-grained clustering of words based on (a proxy measure for) contextual similarity (or rather: an embedding in a high-dimensional space, which implicitly allows the words to be more easily clustered). And yes, some additive relations b…

I wonder if anyone has ever run this system on Lewis Carrol's Jabberwocky[1] or even something like Anthony Burgess's A Clockwork Orange, both of which contain a large number of made up words/slang/re-use.

I remember that when I first read A Clockwork Orange, it took me a while but I finally started to understand the meanings of those words/phrases (though I may not have every encountered them before.) It did feel like my brain was re-wiring itself to a new language. It'd be interesting to see how some type of language AI would treat these works.

https://en.wikipedia.org/wiki/Jabberwocky

edited to add there's a wiki article on the language of A Clockwork Orange, Nadsat: https://en.wikipedia.org/wiki/Nadsat

Re: AI’s Language Problem

#142

Post by xj9 »

digitalarborist wrote: post #97
erikpukinskis wrote: post #84

No one would ever imagine that locking a baby in a featureless room with a giant stack of books would give them general intelligence. I don't understand why AI researchers think it will work for AIs. They need bodies that are biologically connected with the rest of the biosphere, with an intrinsic biological imperative, if they are ever to understand the world. I'm not saying they have to be exactly like us, but they…

It's probably a mistake to assume that just because it's the way we do it that it has to be the way machines do it. Although that's usually the initial assumption. In the early days of flight most attempts were based on birds, similarly submersible vehicles were based on fish. We know now it's better to use propellers. It could be we just haven't found what is analogous to a propeller for the AI world.

The only general intelligence we know of is us. It stands to reason that the first step towards creating AGI is to copy the one machine we know is capable of that type of processing. Why doesn't our research focus on understanding and copying biological brains? Numenta did, with good results, but it isn't an industry trend.

Re: AI’s Language Problem

#143

Post by Joof »

Houshalter wrote: post #102
erikpukinskis wrote: post #84

No one would ever imagine that locking a baby in a featureless room with a giant stack of books would give them general intelligence. I don't understand why AI researchers think it will work for AIs. They need bodies that are biologically connected with the rest of the biosphere, with an intrinsic biological imperative, if they are ever to understand the world. I'm not saying they have to be exactly like us, but they…

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vecto…

The problem seems to be in understanding the overall meaning over long periods of time. We can understand general structure quite easily and a single sentence might make sense, but overall it never ends up cohesive or meaningful.

Maybe we can do that with just words, but humans certainly don't. Words are related to concepts first, then we figure out the meaning of the rest.

Re: AI’s Language Problem

#144

Post by Cybiote »

Houshalter wrote: post #102
erikpukinskis wrote: post #84

No one would ever imagine that locking a baby in a featureless room with a giant stack of books would give them general intelligence. I don't understand why AI researchers think it will work for AIs. They need bodies that are biologically connected with the rest of the biosphere, with an intrinsic biological imperative, if they are ever to understand the world. I'm not saying they have to be exactly like us, but they…

I believe we can get AI from just text. Obviously that won't work for babies, because babies get bored quickly looking at text. AIs can be forced to read billions of words in mere hours! Look at word2vec. By using simple dimensionality reduction on the frequency that words that occur near to each other in news articles, it can learn really interesting things about the meaning of words. The famous example is the vecto…

I agree that AI is possible from text alone but only with the added stipulation that full understanding of text requires a very sophisticated learner. In order to predict what a human will say next, you need the ability to maintain extended context, use world models plus deduction to narrow and maintain multiple possibilities; all while being able to infer the state of the thing you are trying to predict (which means the AI itself has complex internal state far beyond anything an RNN or Neural Turing machine could manage today).

If I said "That is one huge fan", could you predict what my next word will be? It would depend a lot on context and the ability to reason within a complex world model. Depending on whether I had gone to a concert or to a wind tunnel, your distribution over guesses would alter. If I had gone whale watching you might even suspect I made a typo. Changing huge to large would lead to major to no adjustments, depending on each guess.

So while I agree an AI could emerge from text alone, it would have to be very sophisticated to do this.

Re: AI’s Language Problem

#145

Post by ifdefdebug »

sapphireblue wrote: post #44
brianchu wrote: post #8

Deep learning has succeeded tremendously with perception in domains that tolerate lots of noise (audio/visual). Will those successes continue with perception in domains that are not noisy (language) and inference/control , which the article touches on? I think it really is unclear whether those challenges will require fundamental developments or just more years of incremental improvement. If fundamental developments…

It is already succeeding on language tasks, see https://research.facebook.com/research/babi/ It is funny how every AI post on HN turns into a speculative discussion forum full of words "I think", "likely", "I suspect", "My guess" etc, when all the research is available for free and everyone is free to download and read it to get a real understanding of what's going on in the field. >what I see is the fact that networ…

> It is funny how every AI post on HN turns into a speculative discussion forum full of words "I think", "likely", "I suspect", "My guess" etc

This is probably due to the fact that the field is very interesting and has lots of undefined boundaries, so people like to take educated guesses based on the knowledge they might have and on their intuition. Fair enough for this discussion.

> maybe language understanding will be solved in the coming years

maybe? :)

OK, here comes my guess: I think reasoning about and producing computer programs should be easier than reasoning about and producing natural language. So if that's possible (big if), then it should come first. And then maybe the NLP will be solved with the help of code writing computers. Or maybe just by code writing computers, and nobody here has a job anymore :)

Re: AI’s Language Problem

#146

Post by Joof »

Houshalter wrote: post #132
mcphage wrote: post #131

Earlier quoted context omitted.

> What is the meaning of a word, if not the relationship it has to other words? There's also the relationship it has with the world.

Well in my example the AI doesn't have to interact with the world at all. To pass the Turing test simply requires imitating a human, predicting what words they would say. You only need to know the relationships between words.

A beautiful rainbow glimmering gently in the sky after a summer shower.

What do you see? What do you smell? What do you hear? What does the landscape look like? What memory does this bring up?

These are messages that the language is communicating. If an AI can't understand at least some of the content of the message then can it compose one effectively? I'm not certain it can understand the meaning from words alone, but we can certainly try.

Re: AI’s Language Problem

#147

Post by IanCal »

djsumdog wrote: post #81
tormeh wrote: post #71

Earlier quoted context omitted.

No, you only need context. Context in the form of knowledge about the place, company and history that the statement is spoken in. Wikipedia will serve well for a lot of that.

But that's the thing with AI. We make the context. In the case of AlphaGo, IBM's Watson, Self driving cars, we set the goal. There are different heuristics, but we always need to define what is "right" or what the "goal" is. For AI to determine their own goals, well now you get into awareness ... consciousness. At a fundamental mathematical level, we still have no idea how these work. We can see electrical signals in…

> we still have no idea how these work.

I'd actually just say that we've not really defined these very well, and so arguing about how far along the path we are to them isn't that productive.

Re: AI’s Language Problem

#148

Post by Houshalter »

dmreedy wrote: post #134
Houshalter wrote: post #133

Earlier quoted context omitted.

>There is a class of functions that are simply not learnable without some prerequisite knowledge. This is directly analogous to a one-time pad in crypto. It is entirely possible that the function 'language' is in this class of unlearnable functions. I don't understand how this could possibly be the case. We can already make great progress on language understanding with simple methods like word2vec, or perhaps even ma…

It can be the case if Chomsky was right, and Universal Grammar and other similar structures are a thing. That would mean that part of our ability to understand language comes from the particular structure of our brain (which everyone seems to by and large share). That would mean that some of our ability to understand language is genetic in nature, by whatever means genes direct the structure of brain development.

But I don't see any reason a "universal grammar" couldn't be learned. It may take something more complicated than ANNs, of course. But it would be really weird if there was a pattern in language that was so obfuscated it couldn't be detected at all.

Re: AI’s Language Problem

#149

Post by bweitzman »

dmreedy wrote: post #134
Houshalter wrote: post #133

Earlier quoted context omitted.

>There is a class of functions that are simply not learnable without some prerequisite knowledge. This is directly analogous to a one-time pad in crypto. It is entirely possible that the function 'language' is in this class of unlearnable functions. I don't understand how this could possibly be the case. We can already make great progress on language understanding with simple methods like word2vec, or perhaps even ma…

It can be the case if Chomsky was right, and Universal Grammar and other similar structures are a thing. That would mean that part of our ability to understand language comes from the particular structure of our brain (which everyone seems to by and large share). That would mean that some of our ability to understand language is genetic in nature, by whatever means genes direct the structure of brain development.

So if language comes from the structure of the brain, what would stop us from simulating that structure to give a machine mastery of language? And specifically what would imply that a machine which had some of that structure would need to learn by interaction as the top level comment suggests?

Re: AI’s Language Problem

#150

Post by IanCal »

dmreedy wrote: post #33
Retric wrote: post #27

Earlier quoted context omitted.

Language is noisy. People often say things that have little to do with what they mean and context is really important. EX: "How long do stars last?" Means something very different in a science class than a tabloid headline. Is that tabloid talking divorce or obscurity? Notice how three sentences in I am clarifying last .

Yep. The problem is that it's _so_ noisy, that the encryption, as it were, might be too strong to crack with statistical methods. You might need the key; i.e., something like a human brain. EDIT: a combination of noise, I should say, and paucity of information.

Well, we also get things wrong all the time. We regularly either ask for further information to decide what they mean, or expect that it's OK to get the interpretation wrong but be corrected.

Asking a computer to solve all the ambiguity in human language perfectly is asking it to solve it far better than any human can.

Post reply on HN