Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI’s Language Problem

technologyreview.com

Post reply on HN
Print view View original

51–60 of 244 posts

Re: AI’s Language Problem

#51

Post by visarga »

dmreedy wrote: post #19
grondilu wrote: post #5

It seems to me that a full mastery of language requires a grasp of semantics, that is the ability to understand what a sentence means. I doubt it's possible to do that without having basic common sense along with an overall representation of the world, and that looks very close to strong AI, imho. So I'm not surprised computers keep on struggling with language applications. Once they succeed strong AI will not be muc…

I think the 'overall representation of the world' requirement is pretty key here. Language in AI is often treated as its own class of problem, with the assumption that there is somehow enough signal in the raw mess of examples provided to any given learning system (usually just plain text, stripped of any prosody, emotion, cultural context, imagery; any of the other modalities of communication available to a demonstr…

Grounding language in other sense modalities (multimodal learning) is a thing. We can even generate captions from images and generate images from captions, albeit, not perfectly.

Another grounding source is related to ontologies. We are already building huge maps of facts about the world like "object1 relation object2".

Another source of "common sense" is word embeddings. In fact it is possible to embed all kinds of things, like, shopping bags, music preferences, networks topologies - as long as we can observe objects in context.

Then there is unsupervised learning from video and images. For example, starting from pictures, cut them in a 3x3 grid, shuffle the tiles and then task the network to recover the original layout. This automatically extract semantic information from images unsupervised. A variant is to take slides from video, shuffle them around, then task the network to recover the original temporal order. Using this process we can cheaply learn about the world and provide this knowledge as "common sense" for NLP tasks.

I am not worried about grounding language. We will get there soon enough, but we're just impatient. Life evolved over billions of years, AI is just emerging now. Imagine how much computing power is in the collected brains of humanity, and how much computer time we give AI to learn. AI is starved of raw computing power and experience yet. Human brains would have done much worse with the same amount of computing.

Re: AI’s Language Problem

#52

Post by sapphireblue »

BenoitEssiambre wrote: post #43
grondilu wrote: post #5

It seems to me that a full mastery of language requires a grasp of semantics, that is the ability to understand what a sentence means. I doubt it's possible to do that without having basic common sense along with an overall representation of the world, and that looks very close to strong AI, imho. So I'm not surprised computers keep on struggling with language applications. Once they succeed strong AI will not be muc…

Yup, these type of problems have been named AI-Complete ( https://en.wikipedia.org/wiki/AI-complete )

IMHO It is unproductive to rigidly split problems into "merely requiring algorithmic solution" and "AI-complete".

Even with language there is a whole spectrum of language skill. Some animals like parrots, crows, great apes, and then people can learn language at various levels.

Some deep learning models can already learn basic language skills too. The question is, how far can these techniques go. Maybe, pretty far.

Re: AI’s Language Problem

#53

Post by s3arch »

We might still have to go much more basic.. Or we might end up going there in future.... Some have mentioned here having consciousness is more fundamental than language.

Even very basic creatures with less intelligence "learn" because they "want to live".

That is the key - You want to stay alive.

You can't be immortal. You just don't live to learn forever. You live to stay alive and feel happy. And that is what drive us to learn.

Could this be true in case of machines?

Re: AI’s Language Problem

#54

Post by wildmusings »

king_magic wrote: post #37
wildmusings wrote: post #25

Consciousness is the secret sauce. Consciousness as in "subjective experience", what separates us from philosophical zombies, the sensation of the color blue, of middle-c. Our bodies evolved and kept this extremely rich phenomenon for a reason; it is extraordinarily unlikely for it to have arisen and remained through genetic drift alone. My theory, and I'd love to find someone offering a similar and more fleshed-out…

I've had somewhat similar thoughts, and I am entirely unqualified (and highly likely not the first) to put forth the idea that consciousness's killer app is the ability to rapidly assembly abstract models of experiences (present from current sensory input, past from short/long-term memory or future from mental simulation) and be able to query/manipulate those models, and my (admittedly potentially naive) suspicion is…

One possibility about the machinery of consciousness that would help address the Hard Problem is that the brain is not creating consciousness from scratch, but tapping into some currently misunderstood or unknown physical phenomenon. It's hard to see how information processing alone (which can be done by monks with paper and pencil, if incredibly slowly) can give rise to subjective experience.

It seems that this phenomenon, whatever it is, plays a central role in sensory perception, and there's reason to think that it's present even in animals with simple brains. So I suspect that we're looking for some kind of simple operation that can happen on the scale of a small number of neurons, maybe even a single neuron.

This is all speculation of course, informed by some knowledge and intuition, but speculation nonetheless. But it's the only way to push the frontier, and the unwillingness to engage with consciousness as a matter of serious study seems to be a major failing of brain science and AI.

Re: AI’s Language Problem

#55

Post by inputcoffee »

tim333 wrote: post #45
inputcoffee wrote: post #14

Earlier quoted context omitted.

As I understood this article, they're not even trying to. Their version of understanding isn't of the right sort.

I haven't heard of an AI system understanding that yet but it's a fairly common concept in everyday life for instance dogs tend to have masters. Anyone trying to make an AI understand everyday life or literature is going to have to deal with that.

I think the article speaks to this very point. The people "trying to make an AI understand" are not making it "understand" in the sense that you and I use most commonly use the term.

Re: AI’s Language Problem

#56

Post by tim333 »

anonymousguy wrote: post #41

Small child have to learn language from nothing. They just figure it out through exposure and practice. Even pets learn some language. This is the model to emulate. Ultimately language use requires a few skills: * a good parser * motor cognition/coordination * a good memory * semantics/context * vocabulary * situational awareness The first two in the list are what small children struggle with the most. Fortunately, w…

It depends a bit on what you are trying to achieve but I think hooking neural type networks together to simulate human mental faculties might be a better way forward. For instance much of human thinking seems to work around visualizing things in 3d space so you can say to someone imagine a dog on a skateboard on top of a hill and you give it a push, what happens? Once you've got that kind of stuff working with spatial awareness, cause and effect and so on using neural type processing I think the language understanding would come fairly naturally.

Re: AI’s Language Problem

#57

Post by ArkyBeagle »

brianchu wrote: post #8

Deep learning has succeeded tremendously with perception in domains that tolerate lots of noise (audio/visual). Will those successes continue with perception in domains that are not noisy (language) and inference/control , which the article touches on? I think it really is unclear whether those challenges will require fundamental developments or just more years of incremental improvement. If fundamental developments…

But there isn't a model ( that I am aware of, and I've done some serious checking because reasons :) ) goes beyond say, Chomsky saying "we dunno". The difference between human language capability and our nearest evolutionary neighbors is profound, and we appeal to some emergent phenomenon.

But something about the very use of hierarchy in trying to solve NLP makes me queasy. I think it's more (poetically-metaphorically) like Reed-Solomon codes than hierarchies ( to the extent that those don't actually overlap ). There is Unexplained Conservation of Information That Really Isn't There To Start With.

Re: AI’s Language Problem

#58

Post by sapphireblue »

brianchu wrote: post #49
sapphireblue wrote: post #44

Earlier quoted context omitted.

It is already succeeding on language tasks, see https://research.facebook.com/research/babi/ It is funny how every AI post on HN turns into a speculative discussion forum full of words "I think", "likely", "I suspect", "My guess" etc, when all the research is available for free and everyone is free to download and read it to get a real understanding of what's going on in the field. >what I see is the fact that networ…

I specifically and mindfully added those words because everything is really an open research question . Would you rather I dissembled a false sense of confidence? If anything, you're stating your vague case way over-confidently. Turing-completeness is broad and nonspecific. Doing "some computation" is an obvious statement that doesn't add any information. The human brain does not seem to have time limits when it come…

I agree to your points. Your comment is a quality one, I mostly talked about other ones.

>Turing-completeness is quite broad and nonspecific, like I said.

It is, but feedforward models (and almost every Bayesian/statistical model) don't possess it even in theory, while RNNs do.

>Doing "some computation" is an obvious statement that doesn't add any information.

Let me be more specific: currently researchers think that later stages of CNNs do something that is more interpretable as computation than as mere pattern matching. Our world doesn't require 50-level hierarchy, but resnets with 50+ layers do good, looks like because they learn some non-trivial computation.

>the jury is still out on whether any of those RNN approaches will be the needed breakthrough.

Sure, we'll see. Maybe there won't be need in any breakthrough, just incremental improvement of models. And even current models when scaled up to next-gen hardware (see nervana) can surprise us again with their performance.

Re: AI’s Language Problem

#59

Post by sapphireblue »

s3arch wrote: post #53

We might still have to go much more basic.. Or we might end up going there in future.... Some have mentioned here having consciousness is more fundamental than language. Even very basic creatures with less intelligence "learn" because they "want to live". That is the key - You want to stay alive. You can't be immortal. You just don't live to learn forever. You live to stay alive and feel happy. And that is what drive…

This is not a question of "consciousness", this is a question of Reinforcement Learning.

Re: AI’s Language Problem

#60

Post by [deleted] »

sapphireblue wrote: post #58
brianchu wrote: post #49

Earlier quoted context omitted.

I specifically and mindfully added those words because everything is really an open research question . Would you rather I dissembled a false sense of confidence? If anything, you're stating your vague case way over-confidently. Turing-completeness is broad and nonspecific. Doing "some computation" is an obvious statement that doesn't add any information. The human brain does not seem to have time limits when it come…

I agree to your points. Your comment is a quality one, I mostly talked about other ones. >Turing-completeness is quite broad and nonspecific, like I said. It is, but feedforward models (and almost every Bayesian/statistical model) don't possess it even in theory, while RNNs do. >Doing "some computation" is an obvious statement that doesn't add any information. Let me be more specific: currently researchers think that…

[deleted]
Post reply on HN