Live data from Hacker News

AI’s Language Problem

technologyreview.com

41–50 of 244 posts

Re: AI’s Language Problem

#41
Small child have to learn language from nothing. They just figure it out through exposure and practice. Even pets learn some language. This is the model to emulate.

Ultimately language use requires a few skills:

* a good parser * motor cognition/coordination * a good memory * semantics/context * vocabulary * situational awareness

The first two in the list are what small children struggle with the most. Fortunately, we can eliminate motor coordination as a need for AI. Although extremely powerful parsers demand a specialized expertise to produce this part of the problem is straight forward. I write open source multi-language/multi-dialect parsers as an open source hobby.

I discount vocabulary and situational awareness, because most children still haven't figure this out until they enter high school long after they have learned the basics of speech. That pattern of human behavior dictates that while it might be hard to teach these skills to a computer you can put this off a long ways down the road until after basic speech is achieved.

If somebody paid me the money to do this research my personal plan of attack would be:

1. Focus on the parser first. Start with a text parser and do audio to text later. Don't worry about defining anything at this stage. When humans first learn to talk and listen they are focusing upon the words and absolutely not what those words mean.

The parser should not be parsing words. Parsing words from text is easy. The parser should be parsing sentences into grammars, which is harder but still generally straight forward with many edge cases.

2. Vocabulary. Attempt to define words comprising the parsed grammar. Keep it simple. Don't worry about precision at first. Humans don't start with precision and humans get speech wrong all the time. This especially true for pronouns. Just provide a definition.

3. Put the vocabulary together with the parsed grammar. It doesn't even have to make sense. It just has to have meaning for words and the words together in a way that informs an opinion or decision to the computer. Consider this sentence as an example: I work for a company high up in the building with a new hire that just got high and gets paid higher than my high school sweetheart.

4. If the sentence is part of a paragraph or a response to a conversation you can now focus on precision. You have additional references from which to draw upon. You are going to redefine some terms, particularly pronouns. Using the added sentences make a decision as to whether new definitions apply more directly than the original definitions. This is how humans do it. These repeated processing steps means wasted CPU cycles and its tiring for humans too.

5. Formulate a response. This could be a resolution to close the conversation, or it could be a question asking for additional information or clarity. Humans do this too.

6. Only based upon the final resolution determine what you have learned. Use this knowledge to make decisions to modify parsing rules and amend vocabulary definitions. The logic involved is called heuristics.

This only way all this works is to start small, like a toddler, and expand it until the responses become more precise, faster, and more fluid. At least.... this is how I would do it.

Re: AI’s Language Problem

#42
post #17

Earlier quoted context omitted.

Nor does a meteor know what it means to be an extinction event.

True but a meteor doesn't have to. As it has been written of in our sci-fi, the robot uprising specifically requires the machines to understand and, most of all, care, about dominance. If you're talking about just being victims of machine logic, we've been suffering that since the invention of the traffic light traffic jam.

It may not need to understand or care. It may just be optimizing towards a goal, which involves 'humans need to die' or 'destroy/pollute the water supply', etc. It may indirectly cause serious damage without even awareness we exist.

Re: AI’s Language Problem

#43
post #5

It seems to me that a full mastery of language requires a grasp of semantics, that is the ability to understand what a sentence means. I doubt it's possible to do that without having basic common sense along with an overall representation of the world, and that looks very close to strong AI, imho. So I'm not surprised computers keep on struggling with language applications. Once they succeed strong AI will not be muc…

Yup, these type of problems have been named AI-Complete ( https://en.wikipedia.org/wiki/AI-complete )

Re: AI’s Language Problem

#44
post #8

Deep learning has succeeded tremendously with perception in domains that tolerate lots of noise (audio/visual). Will those successes continue with perception in domains that are not noisy (language) and inference/control , which the article touches on? I think it really is unclear whether those challenges will require fundamental developments or just more years of incremental improvement. If fundamental developments…

It is already succeeding on language tasks, see https://research.facebook.com/research/babi/

It is funny how every AI post on HN turns into a speculative discussion forum full of words "I think", "likely", "I suspect", "My guess" etc, when all the research is available for free and everyone is free to download and read it to get a real understanding of what's going on in the field.

>what I see is the fact that networks have huge capacity to fit to data and are deep (rely on a hierarchy of features).

Actually recurrent neural networks like LSTM are turing-complete, i.e. for every halting algorithm it is trivial to implement an RNN that computes it. It is non-trivial to learn these parameters from algorithm IO data, but for many tasks it is possible too.

>I suspect that most of what is being learned (and solved) is perception of useful features from the raw game images.

It is not this simple, deep enough convnets can represent computations, the consensus is that middle and upper layers of convnets represent some useful computation steps. Also note that human brain can only do so much computation steps to answer questions when in dialogue, due to time and speed limits.

>My guess is we may need a fundamental breakthrough in a newfangled hierarchical learning system that is better suited for language to “solve” NLP.

This is being worked on, see the first link for Memory Networks and Stack RNNs, DeQue RNNs, Tree RNNs. Deep learning is a very generic term, there are dozens of various feedforward and recurrent architectures that are fully differentiable. The full potential of such models has not been nearly reached yet and maybe language understanding will be solved in the coming years (again, the first link shows that it is in process of being solved).

Re: AI’s Language Problem

#45
post #12

Earlier quoted context omitted.

Not yet.

As I understood this article, they're not even trying to. Their version of understanding isn't of the right sort.

I haven't heard of an AI system understanding that yet but it's a fairly common concept in everyday life for instance dogs tend to have masters. Anyone trying to make an AI understand everyday life or literature is going to have to deal with that.

Re: AI’s Language Problem

#46
post #26
post #8

Deep learning has succeeded tremendously with perception in domains that tolerate lots of noise (audio/visual). Will those successes continue with perception in domains that are not noisy (language) and inference/control , which the article touches on? I think it really is unclear whether those challenges will require fundamental developments or just more years of incremental improvement. If fundamental developments…

I think you're dead on. And I'm nervous about a coming winter, because of disappointment in all the wolf-crying we're doing about how good at Natural Language we're getting. When we've barely scratched the surface. This latest bot fad worries me. A further comment on deep methods being state of the art currently: I wonder how well these tasks really measure progress in natural language understanding (I really don't l…

Chatbots aren't about nlp, IMHO, they are about easing non-technical people into a top level cli for everything. IMHO, they ultimately have as much in common with search as with a natural language interaction.

IME as a chatbot developer, people don't talk to them in conversational english so much as spit out what they want it to do.

Re: AI’s Language Problem

#47
post #39

Solving language for computers seems much like climbing a series of mountains, where each time you surmount one, you realize the next is even higher :) Thanks to deep learning, machines have made rapid gains in speech recognition, as well as improving semantic mapping (a la word2vec and other word embedding approaches). But once you have a system with human-level speech recognition and semantic mapping, where do we g…

"much like climbing a series of mountains, where each time you surmount one, you realize the next is even higher"

Aren't you describing learning, in general? Physics, math, biology, etc

Re: AI’s Language Problem

#48

If we use animals as a reference, I would say that consciousness is more fundamental than language, so most likely we need that in place before we can get AI to be able to effectively understand language.

[deleted]

Re: AI’s Language Problem

#49
post #8

Deep learning has succeeded tremendously with perception in domains that tolerate lots of noise (audio/visual). Will those successes continue with perception in domains that are not noisy (language) and inference/control , which the article touches on? I think it really is unclear whether those challenges will require fundamental developments or just more years of incremental improvement. If fundamental developments…

It is already succeeding on language tasks, see https://research.facebook.com/research/babi/ It is funny how every AI post on HN turns into a speculative discussion forum full of words "I think", "likely", "I suspect", "My guess" etc, when all the research is available for free and everyone is free to download and read it to get a real understanding of what's going on in the field. >what I see is the fact that networ…

I specifically and mindfully added those words because everything is really an open research question. Would you rather I dissembled a false sense of confidence? If anything, you're stating your vague case way over-confidently. Turing-completeness is broad and nonspecific. Doing "some computation" is an obvious statement that doesn't add any information. The human brain does not seem to have time limits when it comes to thinking about what to say, and further we don't understand enough about neuroscience to make statements like that. Like I said, these are all active areas of research; the jury is still out on whether any specific approach will be the breakthrough.

EDIT (reply to below): in general these statements are either vague and nonspecific, or perfectly correct and non-informative, comments that don't have much to do with my original point.

Re: AI’s Language Problem

#50

I think about the AI language problem a lot while raising my kids. The article notes the word "forever" and how an AI must distinguish the literal from the figurative meaning of the word in context. My five-year-old still doesn't grasp the literal meaning of this word as "never-ending." To him, "forever" is simply a very very long time. He has the same problem with the concept of "infinity," where the word means both…

>Are the image-recognition AI's described in the article capable of selective attention?

Yes they are: https://indico.io/blog/sequence-modeling-neural-networks-par... https://github.com/harvardnlp/seq2seq-attn

Post reply on HN