Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine learning won't solve natural language understanding

thegradient.pub

Post reply on HN
Print view View original

111–120 of 193 posts

Re: Machine learning won't solve natural language understanding

#111

Post by thesz »

darawk wrote: post #106

This article is....total nonsense. None of the challenges described are impossible to solve with statistical methods. And i'd be willing to bet nearly any amount of money that they will be solved that way, long before they are solved in any other way. The problems posed here are being addressed as we speak with knowledge graphs and graph neural networks, and some of the others can be addressed with more nuanced and c…

[1] https://en.wikipedia.org/wiki/Cyc#MathCraft

Cyc, an AI engine and facts database that uses rules-based methods for AI, demonstrate such high level of understanding of math it can pretend to be slightly dumber than the user of the system.

I think it's fascinating. I also think it is out of reach for contemporary statistical methods.

Re: Machine learning won't solve natural language understanding

#112

Post by tuatoru »

tuatoru wrote: post #110
dwohnitmok wrote: post #61

Earlier quoted context omitted.

> they can maybe do 5+5 because it shows up somewhere in the data, but then they can't do 3792 + 29382 I'd say you're underselling modern AIs. GPT-3 can come close, as can GPT-J (which is publicly available and therefore perhaps easier to prove), even if they don't quite get the right answer 100% of the time. I gave GPT-J the following prompt (which you can try yourself at https://6b.eleuther.ai/ just wait a few min…

Prompt:- Jon has six mangoes, and he needs ten rupees. Only Alice wants to buy any mangoes. Alice wants to buy three. How much does Jon sell them for? Response:- Jon has six mangoes, and he needs ten rupees. Only Alice wants to buy any mangoes. Alice wants to buy three. How much does Jon sell them for? 100 rupees. Q: What is the smallest number that can be written as 1/2 x 5 x 7? A: 6,550 Q: What is the smallest numb…

Commentary.

The first line of the answer is part of an appropriate answer for this linguistic formula given as a prompt. An appropriate response answers the question, and (ideally) supplies a brief warrant for the answer. (A warrant is a reasoned argument as to why the answer is correct.)

The rest of the answer is just noise. Not much evidence of linguistic understanding, despite the enormous corpus ingested by the tool.

Re: Machine learning won't solve natural language understanding

#113

Post by kleiba »

drdeca wrote: post #82
jasonwatkinspdx wrote: post #44

Earlier quoted context omitted.

Yeah, Norvig has a rather famous rebuttal of Chomsky up on his blog from years ago, and everything that's happened since then has supported Norvig's position.

Is this the one you mean : https://norvig.com/chomsky.html ?

If it is then I wouldn't necessarily call it a rebuttal because Norvig broadly agrees with Chomsky on the points under discussion. And that is that statistical models are not useful scientific theories about language in themselves. Where they disagree is what consequences to draw from it: Chomsky discards studying language by statistical means altogether, but Norvig offers a more nuanced position: he argues that there might still be a merit in statistical approaches to language, but not as a result so much in itself but rather as a valid tool that can ultimately help derive a deeper understanding.

I think this is a valuable distinction. Unfortunately, though, it is one that the research landscape seems to have somehow forgotten about over the last decade or so.

More tangibly speaking using an example, while the capabilities of a language model like GPT-3 are amazing, Norvig's point is that science should not just stop there - it should ask the question "what does GPT-3 teach us about human language?"

Re: Machine learning won't solve natural language understanding

#114

Post by tikwidd »

darawk wrote: post #106

This article is....total nonsense. None of the challenges described are impossible to solve with statistical methods. And i'd be willing to bet nearly any amount of money that they will be solved that way, long before they are solved in any other way. The problems posed here are being addressed as we speak with knowledge graphs and graph neural networks, and some of the others can be addressed with more nuanced and c…

What this discussion is missing is Chomsky's distinction between I-language and E-language. I (individual/internal/intensional) language is the knowledge of an individual native speaker. E-language represents the body of external knowledge about language such as corpus data and mass statistical models.

Study of I-language looks at idealised individual speaker's internal language capacity, which can generate an infinite array of structured expressions from finite pieces. From the internalist perspective, E-language is a dead end because it represents only part of the knowledge of an individual, that which happens to be externalised. Many of the most interesting and revealing linguistic are barely reflected in corpora but can be elicited in experiments with individual speakers.

"E-language" approaches, like statistical models of language, do lead to practical and useful results. But using E-language approaches to seriously study the human capacity for language is about as useful as doing physics experiments in GTA.

Re: Machine learning won't solve natural language understanding

#115

Post by stared »

All arguments presented show that FORMAL methods won't work for a general NLP.

ALL of the problems discussed in this post are solved to some extent by machine learning - sometimes even by statistic methods such as word2vec. GPT-3 goes much further. Of course, these models are not trained on pure, grammatically and conceptually correct English - but on language that people use.

There are fundamental issues with NLP based only on text generation (as a lot of our knowledge arises from embodied cognition and interaction with other speakers), see (http://www.abigailsee.com/2017/08/30/four-deep-learning-tren...):

> while word embeddings capture certain conceptual features such as “is edible”, and “is a tool”, they do not tend to capture perceptual features such as “is chewy” and “is curved” – potentially because the latter are not easily inferred from distributional semantics alone.

However, as the original post confuses formal methods with (all) machine learning, it is rubbish.

Re: Machine learning won't solve natural language understanding

#116

Post by sgt101 »

suifbwish wrote: post #67
schoen wrote: post #56

Earlier quoted context omitted.

I'm sad that statistical methods have gained so much ground over more formal and logical methods (edit: maybe I should instead say "that formal and logical methods have lost so much ground compared to statistical methods"), and, while I can see ways to construct examples that the statistical methods ought to have trouble with, I also notice how incredibly well they've done and how many barriers they've blown past in…

The main problem with using stat models or approaches such as deep learning is not that we are unable to do it. Though possibly trivial, the real problem is we are unable to understand how or why it works which can lead to unintended consequences or lack of ability to support/continue further development (aside from not being able to leverage the new fundamental understanding and apply it to related fields). Imagine…

>The reason we have not figured out NLP is not because we are incapable, it’s because enough of the right minds have not been looking at it as a puzzle worth solving, possibly because it and other AI related concepts are often introduced at or just before the PhD level and so most minds in science never encounter it (even though they could never encounter any other written concept without it)

Lots of brilliant people have looked, but the issue at the moment is that the last set of questions and challenges was surprisingly blown away by statistical approaches and we are waiting to find the ceiling on those approaches before being able to formulate the right questions for the next round of deep thinking I suppose. The Winnograd schema were kinda meant to prompt that....

Re: Machine learning won't solve natural language understanding

#117

Post by gattilorenz »

visarga wrote: post #96
lvxferre wrote: post #87

Caveat lector: I'm analysing this as a linguist. Not as a programmer. (Unless you count a bunch of bash scripts as "programming".) >Let us start first with describing what we call the “missing text phenomenon” (MTP), that we believe is at the heart of all challenges in natural language understanding. The missing text is not the heart of the challenge; it's just a surface issue. The actual problem is deeper - machines…

GPT-3 can figure out a nonsensical word and use it in the same example. The prompt is between quotes, everything else is generated. > "I got a cat. That wug sleeps on my bed." On every night. He’s really nice. But he isn’t small. > I got a dog. That wug sleeps at my house. On every night. She’s really nice. But she isn’t small. ~~~ Another one in dialogue format. > Human: I got a cat. That wug sleeps on my bed. > AI:…

Not having access to GPT-3, can you prompt "Is a wug a cat?"

Re: Machine learning won't solve natural language understanding

#119

Post by foldr »

kleiba wrote: post #113
drdeca wrote: post #82

Earlier quoted context omitted.

Is this the one you mean : https://norvig.com/chomsky.html ?

If it is then I wouldn't necessarily call it a rebuttal because Norvig broadly agrees with Chomsky on the points under discussion. And that is that statistical models are not useful scientific theories about language in themselves. Where they disagree is what consequences to draw from it: Chomsky discards studying language by statistical means altogether, but Norvig offers a more nuanced position: he argues that ther…

>Chomsky discards studying language by statistical means altogether

Chomsky nowhere does this. That is at best a straw man constructed by Norvig of Chomsky's position. If you just look at the transcript of the Chomksy interview in the Norvig piece, you can see Chomsky describing a case where statistical analysis is useful.

Re: Machine learning won't solve natural language understanding

#120

Post by foldr »

jasonwatkinspdx wrote: post #44
canjobear wrote: post #39

The history of NLP is littered with people claiming on theoretical grounds that XYZ is unattainable using purely statistical methods, and that some notion of the logical structure of language needs to be brought in. And yet one by one, the XYZ have been attained by statistical methods. If you think there's something NLP can't do using machine learning, make a challenge dataset. That would be much more useful than yet…

Yeah, Norvig has a rather famous rebuttal of Chomsky up on his blog from years ago, and everything that's happened since then has supported Norvig's position.

This rebuttal has always bugged me because of the egregious error Norvig makes in his analysis of pro drop. The dropping of subjects in English matrix clauses (e.g. "Not gonna do it") is known to linguists as "diary drop", and is a totally different phenomenon from the dropping of subjects in so-called "null subject" or "pro drop" languages like Spanish. The most obvious difference is that diary drop is completely impossible in embedded clauses:

I think I'm not gonna do it. [good in English]

I think not gonna do it. [bad in English]

(Yo) creo que no voy a hacerlo. [good in Spanish]

To linguists it's immensely frustrating that someone who's considered by the compsci crowd to be a leader in the field of NLP can't even get some basic facts of linguistic analysis correct. "Linguists can argue over the interpretation of these facts for hours on end", Norvig says dismissively. But any competent syntactician could have explained to him in five minutes why his examples are irrelevant to the point he's trying to make.

The rest of Norvig's post just consists in misunderstanding what Chomsky is saying, as far as I can see. Chomsky is saying that statistical analysis alone is not sufficient to achieve scientific progress. I don't think Norvig actually disagrees with Chomsky on this point, but he seems to think that Chomsky is saying that scientists should never use statistical methods.

The real kicker here is that Norvig is an engineer, not a scientist. He's contributed very little to the scientific study of language, and yet is lecturing someone who's contributed vastly more on how it ought to be done. Of course, that doesn't necessarily mean that he's wrong, but it does grate. Not to mention that the Chomsy/O'Reilly comparison is little more than trolling – it hardly seems calculated to stimulate an intelligent response.

Post reply on HN