Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning won't solve natural language understanding

thegradient.pub

Post reply on HN
Print view View original

121–130 of 193 posts

Re: Machine learning won't solve natural language understanding

#121

Post by PeterisP »

lvxferre wrote: post #87

Caveat lector: I'm analysing this as a linguist. Not as a programmer. (Unless you count a bunch of bash scripts as "programming".) >Let us start first with describing what we call the “missing text phenomenon” (MTP), that we believe is at the heart of all challenges in natural language understanding. The missing text is not the heart of the challenge; it's just a surface issue. The actual problem is deeper - machines…

> "I dropped the trophy on the table. Now it's broken." (What's broken - the trophy, or the table?)

We could not solve this kind of problems (Winograd schemas) automatically a few years ago, but now ML techniques can solve them at about the same error rate as humans.

Re: Machine learning won't solve natural language understanding

#123

Post by ZephyrBlu »

exporectomy wrote: post #11

I didn't get beyond his argument that ML is compression while NLU is decompression but that doesn't seem to be right. ML is often used to decompress data such as increasing image resolution. Of course it needs more data than just the compressed form, but only for training. For inference, of course you can use ML to add assumed common knowledge information.

How is ML not compression?

You take a training set, and you end up with a model that is smaller than the size of the training set yet performs well on it (Making the assumption the model is good here).

That is compression. It has nothing to do with what the model is being used for.

Re: Machine learning won't solve natural language understanding

#124

Post by slunk »

exporectomy wrote: post #11

I didn't get beyond his argument that ML is compression while NLU is decompression but that doesn't seem to be right. ML is often used to decompress data such as increasing image resolution. Of course it needs more data than just the compressed form, but only for training. For inference, of course you can use ML to add assumed common knowledge information.

I was hoping there would be discussion of this point higher up in the thread, because I had essentially the same reaction as you while reading this passage. I'm no expert of machine learning, NLP, or linguistics, but this struck me as a pretty obvious flaw in the author's argument.

Re: Machine learning won't solve natural language understanding

#125

Post by evrydayhustling »

joe_the_user wrote: post #59
evrydayhustling wrote: post #25

The author seems hung up on the idea that because NLU involves apparent "discontinuities" -- places where small variations in interpretation completely transform tasks -- it will not be amenable to smooth, continuous notions like PAC learning, compressibility, and so on. While there's a directional insight there, the terms aren't well defined. And the big story of ML-based NLP in the last decade has been that many ta…

And the big story of ML-based NLP in the last decade has been that many tasks that were presumed too jagged for curve fitting are in fact tractable... The thing is, I see many situations where large language models have been "impressive" but few situation where they have clearly succeeded in the real world. I'm less than impressed with online translation, Google's AI search is annoying, GPT-3 authored articles are im…

It's not just benchmark tasks, but I think perceptions of progress get skewed because we're in an uncanny period where NLP is good enough for many industrial applications but unsatisfying under detailed scrutiny.

- Online translation? Not for a literary piece, but it will still let your e-commerce site get the gist of a customer complaint.

- Authored text? Not good enough for direct consumption, but pass it through one intern and you get a much faster rate of e.g. satisfactory social media responses.

- Frustrating phone or bot interface? Average customer spends 10% more time, but the company saves 50% of its costs.

Most of these applications transfer some burden downstream, but not all of it... so it is having big impact on the information supply chain. I don't expect those applications too be exciting to many people here, and especially to AGI acolytes, but lots of technologies have gone through this maturity curve: (1) solve toy problems, (2) solve lame but valuable problems, (3) do interesting "real" things.

And in a few places, NLP is moving on to (3). Tools like Grammarly are actually a better experience than most human editing loops. I would also put NLP-backed search in this category -- anyone who Googles is having a much better experience because of modern NLP, without even needing to be aware of it.

Re: Machine learning won't solve natural language understanding

#126

Post by slunk »

ZephyrBlu wrote: post #123
exporectomy wrote: post #11

I didn't get beyond his argument that ML is compression while NLU is decompression but that doesn't seem to be right. ML is often used to decompress data such as increasing image resolution. Of course it needs more data than just the compressed form, but only for training. For inference, of course you can use ML to add assumed common knowledge information.

How is ML not compression? You take a training set, and you end up with a model that is smaller than the size of the training set yet performs well on it (Making the assumption the model is good here). That is compression. It has nothing to do with what the model is being used for.

> It has nothing to do with what the model is being used for.

I may be misunderstanding this passage of the article, but I thought the author was claiming that machine learning (specifically training) was equivalent to compression, while language understanding is equivalent to decompression. Therefore, they can't be the same thing. Why does language understand have to be analogous to training an ML model rather than using an ML model for inference?

Re: Machine learning won't solve natural language understanding

#127

Post by smallerfish »

joe_the_user wrote: post #59
evrydayhustling wrote: post #25

The author seems hung up on the idea that because NLU involves apparent "discontinuities" -- places where small variations in interpretation completely transform tasks -- it will not be amenable to smooth, continuous notions like PAC learning, compressibility, and so on. While there's a directional insight there, the terms aren't well defined. And the big story of ML-based NLP in the last decade has been that many ta…

And the big story of ML-based NLP in the last decade has been that many tasks that were presumed too jagged for curve fitting are in fact tractable... The thing is, I see many situations where large language models have been "impressive" but few situation where they have clearly succeeded in the real world. I'm less than impressed with online translation, Google's AI search is annoying, GPT-3 authored articles are im…

Indeed, translating from the #1 language to the #4 language (or back) is problematic. Google's translation is usually enough to approximate what was said, but often results in gibberish. As a small example, it consistently fails with the spanish pronoun "su" - which is a word that depending on context can mean "his", "hers", "its", "your", "y'all's", "their" -- which seems to be a good example of the compressibility of natural language. How big does the corpus used to train their models need to be to get "su" right?

Re: Machine learning won't solve natural language understanding

#128

Post by im3w1l »

schoen wrote: post #56
canjobear wrote: post #39

The history of NLP is littered with people claiming on theoretical grounds that XYZ is unattainable using purely statistical methods, and that some notion of the logical structure of language needs to be brought in. And yet one by one, the XYZ have been attained by statistical methods. If you think there's something NLP can't do using machine learning, make a challenge dataset. That would be much more useful than yet…

I'm sad that statistical methods have gained so much ground over more formal and logical methods (edit: maybe I should instead say "that formal and logical methods have lost so much ground compared to statistical methods"), and, while I can see ways to construct examples that the statistical methods ought to have trouble with, I also notice how incredibly well they've done and how many barriers they've blown past in…

I think all you need to do is wait. Eventually the statistical methods will stop progressing so quickly. Either because they starting hitting tough obstacles, or because they manage to solve all the problems they set out to solve and go "now what?". That's when interest in alternative methods will pick up.

Re: Machine learning won't solve natural language understanding

#129

Post by smallerfish »

djoldman wrote: post #76

"The man saw a boy with a telescope." There is NO exact interpretation of the above sentence. It is possible that the man peered through a telescope to see the boy, or that the man saw a boy who had a telescope. One may reasonably argue that one interpretation is more likely but either may be true. Ambiguity is an immense problem in semantic analysis, compounded by assumed speaker intent, etc. Ultimately there is no…

Right, but human conversation doesn't work like that. A real usage would be more like "The man saw a boy with his telescope. The boy was running down the beach, apparently screaming out at the sea."

Using "a" in place of "his" in speech would be unnatural and would raise eyebrows. "His" is still ambiguous though, and any human speaker would know that and provide more context. The second sentence disambiguates the sentence sufficiently. Eleuther's online demo still fails badly given the above prompt:

> The boy saw the man, stopped, turned around, and looked at the man angrily, pointing at the horizon. The man saw a break in the waves, and he walked towards the boy. The boy ran into the sea, and the waves tried to take the man, but the man was able to escape from the waves. The boy was angry. He called the man a “nasty pirate.” The man went home and told his mother about the boy and the pirate. The man’s mother said, “He’s just a kid, he probably has no idea what he’s saying.”

Re: Machine learning won't solve natural language understanding

#130

Post by emodendroket »

zR0x wrote: post #92
emodendroket wrote: post #86

Earlier quoted context omitted.

Theoretically, if we had a perfect understanding of human mechanisms of language, we could reproduce it in code. Of course we don't.

I don’t understand how we don’t? I make sound snapping my fingers. Whistling. Why is it so hard to accept mirror neurons fired when early humans heard birds and animals, each other’s grunts and over time we refined it? Everyone has some capacity to refine and strengthen muscle. Why do we need some abstract meta-construct to explain where language comes from? It comes from us. Fleshy meat bags that mutate state over a…

I'm not talking about "where language comes from." I'm talking about a mechanical understanding of how humans produce and understand language.

Chomsky's idea goes deeper than "just random sounds," given his theory of universal grammar.

Post reply on HN