Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Happened to Old School NLP?

languagengine.co

Post reply on HN
Print view View original

31–40 of 49 posts

Re: What Happened to Old School NLP?

#31

Post by ryanmim »

psygnisfive wrote: post #27
ryanmim wrote: post #21

This is a pretty good explanation of why almost all practical applications of NLP are now accomplished by statistics rather than fancy linguistic grammar models you might have read about in a Chomsky book. Old school NLP has always fascinated me though, and I'm pretty excited about what might be possible in the future by using more than purely statistical methods for accomplishing NLP tasks. Maybe the author could ha…

Well, if you'd like to know what I think we'll be doing in the future, check out the rest of the site. :p But: I'm building an SDK for conversational AI (think Siri, in any app, and 10 times better), that's what the site as a whole is for. I think in 5 years it'll be pretty commonplace to have fairly natural, Jarvis-like conversations with computers, and within 10 years we'll have R2D2/C3PO robots.

Ya I checked out your root project, will give it a whirl when you open it up. I'm moderately interested in adding voice commands to an app I'm working on and haven't found a service that fits the bill yet.

Re: What Happened to Old School NLP?

#32

Post by microtonal »

SixSigma wrote: post #22
microtonal wrote: post #7

We'll start to see the re-emergence of tools from old-school NLP, but now augmented with the powerful statistical tools and data-oriented automation of new-school NLP. IBM's Watson already does this to some extent. This is not a new trend. As early as 1997, Steven Abney augmented [1] attribute-value grammars with discriminative modelling (maximum entropy models) in this case to form 'stochastic attribute-value gramma…

The problem is slippery though, people always reuse words to express different ideas : "We can't buy any bread because we haven't got any bread." And it's not just English. In Chinese one is taught that "ni hao ma?" is the greeting equivalent to "hello, how are you" but try it on a Chinese person and it amuses them. My Chinese friend at Uni says that Chinese people use "Ni chi ma?" which is literally "have you eaten?…

> The problem is slippery though, people always reuse words to express different ideas : > "We can't buy any bread because we haven't got any bread."

As long as we are talking about syntactic parsing, this is not a problem as long as the attachment is the same. In both cases 'bread' is the direct object of the main verb.

Of course, there are cases where a particular word can be used both as a direct object and a subject of a particular verb. E.g.:

The man ate the pig.

The pig ate the apple.

Of course, what such systems are learning are not rules, but probability distributions that combine information about the distributions of word orders, association strengths between heads and dependent with a particular dependency relation, configurations of dependent pairs, etc.

Re: What Happened to Old School NLP?

#33

Post by psygnisfive »

ryanmim wrote: post #31
psygnisfive wrote: post #27

Earlier quoted context omitted.

Well, if you'd like to know what I think we'll be doing in the future, check out the rest of the site. :p But: I'm building an SDK for conversational AI (think Siri, in any app, and 10 times better), that's what the site as a whole is for. I think in 5 years it'll be pretty commonplace to have fairly natural, Jarvis-like conversations with computers, and within 10 years we'll have R2D2/C3PO robots.

Ya I checked out your root project, will give it a whirl when you open it up. I'm moderately interested in adding voice commands to an app I'm working on and haven't found a service that fits the bill yet.

Well let me know what sorts of things you have in mind, and what you can't find in other services! I'll see if there's something LE can do for you, or could do in the near future. I'm always in the irc channel (#languagengine on freenode), so feel free to drop by. :)

Re: What Happened to Old School NLP?

#34

Post by microtonal »

SixSigma wrote: post #25
cynicalkane wrote: post #23

Earlier quoted context omitted.

The Taiwanese say "ni hao ma". That's probably where pedagogy got it from.

I imagine plenty of Chinese say it too, really, just like we are taught in my Chinese class at Uni. The point being though, that even if you are taught "Hello, how are you" in an English class then you could well be stumped when you come to my region and we say "How's tricks?"

The referenced papers are about syntactic parsing, not semantics.

Re: What Happened to Old School NLP?

#35

Post by microtonal »

psygnisfive wrote: post #26
JacobiX wrote: post #17

The beginning of the article reminds me of the quote : "Every time I fire a linguist, the performance of our speech recognition system goes up." But nowadays statistical NLP systems regularly use syntactic and semantic information as a features in the learning phase.

It really depends on what you count as syntactic and semantic information. As a linguist, to me syntactic information is tree structures, syntactic categories, etc., and semantics is formulas in some (typically higher-order) logic. But for a lot of the NLP that I see, "syntax" is pretty shallow stuff like head words and POS tag contexts, and "semantics" is at best things like word vectors maybe dependency trees. Thes…

There have been some inflated claims, e.g. people calling their part-of-speech tagger a shallow parser or their shallow parser (e.g. chunking plus some rules) a parser :).

But I think in general computational linguists would say that dependency trees are definitely syntax.

Re: What Happened to Old School NLP?

#36

Post by psygnisfive »

microtonal wrote: post #32
SixSigma wrote: post #22

Earlier quoted context omitted.

The problem is slippery though, people always reuse words to express different ideas : "We can't buy any bread because we haven't got any bread." And it's not just English. In Chinese one is taught that "ni hao ma?" is the greeting equivalent to "hello, how are you" but try it on a Chinese person and it amuses them. My Chinese friend at Uni says that Chinese people use "Ni chi ma?" which is literally "have you eaten?…

> The problem is slippery though, people always reuse words to express different ideas : > "We can't buy any bread because we haven't got any bread." As long as we are talking about syntactic parsing, this is not a problem as long as the attachment is the same. In both cases 'bread' is the direct object of the main verb. Of course, there are cases where a particular word can be used both as a direct object and a subj…

I think they're saying tho that in one case "bread" means a kind of food, and in the other, it means money. If the different uses were more directly connected to particular words that could disambiguate, it wouldn't be too hard -- head features typically can do this -- but the disambiguation here is far more conceptual in original. You know that you need money to buy things, and so you know that not having money is a good reason for not being able to buy things. But bread-the-food is not needed to buy things, so not having it isn't a good reason for not being able to buy things. So probably the second "bread" is the bread-money version. This kind of disambiguation is super tricky without broader world knowledge.

Re: What Happened to Old School NLP?

#37

Post by psygnisfive »

microtonal wrote: post #35
psygnisfive wrote: post #26

Earlier quoted context omitted.

It really depends on what you count as syntactic and semantic information. As a linguist, to me syntactic information is tree structures, syntactic categories, etc., and semantics is formulas in some (typically higher-order) logic. But for a lot of the NLP that I see, "syntax" is pretty shallow stuff like head words and POS tag contexts, and "semantics" is at best things like word vectors maybe dependency trees. Thes…

There have been some inflated claims, e.g. people calling their part-of-speech tagger a shallow parser or their shallow parser (e.g. chunking plus some rules) a parser :). But I think in general computational linguists would say that dependency trees are definitely syntax.

It depends on the kind of dependency trees. I've seen plenty of semantic-y dependency trees and plenty of syntactic-y dependency trees. I'm not sure how common either really are ever, but on the semantics side, it's the best you get, usually, and isn't all that good for semantics. It's fine for syntax, more or less.

Re: What Happened to Old School NLP?

#38

Post by sdoering »

I really liked the article and some of the Blog-Headlines seemed to be interesting as well. But try as I might, I was not able to find a rss/atom/xml-Feed for plugging this ressource into my feedreader. Sadly, so I will probably miss upcoming interesting posts.

Re: What Happened to Old School NLP?

#39

Post by microtonal »

psygnisfive wrote: post #37
microtonal wrote: post #35

Earlier quoted context omitted.

There have been some inflated claims, e.g. people calling their part-of-speech tagger a shallow parser or their shallow parser (e.g. chunking plus some rules) a parser :). But I think in general computational linguists would say that dependency trees are definitely syntax.

It depends on the kind of dependency trees. I've seen plenty of semantic-y dependency trees and plenty of syntactic-y dependency trees. I'm not sure how common either really are ever, but on the semantics side, it's the best you get, usually, and isn't all that good for semantics. It's fine for syntax, more or less.

I think that if you say: dependency treebank, most computational linguists will expect syntactic dependencies. I agree that the notion is blurred sometimes, e.g. by including word senses or preferring semantic attachments in some cases over semantic attachment. There may also be treebanks of the semantic-y kind, but I haven't seen or used them often.

Re: What Happened to Old School NLP?

#40

Post by sqrt17 »

ryanmim wrote: post #21

This is a pretty good explanation of why almost all practical applications of NLP are now accomplished by statistics rather than fancy linguistic grammar models you might have read about in a Chomsky book. Old school NLP has always fascinated me though, and I'm pretty excited about what might be possible in the future by using more than purely statistical methods for accomplishing NLP tasks. Maybe the author could ha…

It's important to make a distinction between (i) Chomskyan linguistics, (ii) 90s style symbolic systems, (iii) 90s/early 2000s style statistical systems and (iv) 2010s style statistical systems.

Chomskyan linguistics assumes that statistics and related stuff is not relevant at all, and that instead you need to find the god-given (or at least innate) Universal Grammar and then everything will be great. 90s style symbolic systems adopt a more realistic approach, relying on lots of heuristics that kind of work but aim at good performance rather than unattainable perfection; 90s style statistical models give up some of the insights in these heuristics to construct tractable statistical models.

If you look at 2010s style statistical models, you'll notice that machine learning has become more powerful and you can use a greater variety of information, either using good linguistic intuitions (which help even more with better learning algorithms, but require a certain expressivity as well as some degree of matching between the way of constructing the features and the classification) or unsupervised/deep-NN learning, which constructs generalizations over features.

The main reason that you won't ever see people talking about systems with great machine learning and great linguistic intuitions is that you normally want to treat one of them as fixed and focus on improving the other, i.e., it's more a practical/cultural difference than an actual limitation.

Post reply on HN