Live data from Hacker News

Python NLTK Bayesian Classifier for word sense disambiguation - 92% accuracy

litfuel.net

21–26 of 26 posts

Re: Python NLTK Bayesian Classifier for word sense disambiguation - 92% accuracy

#21

He did it for one word. Bad article title.

It was actually a method of using wikipedia to build your corpus for any ambiguous word to automatically build some word sense disambiguation in your application. One word was just a simple example of using that data.

- The article does not add anything new. Using Wikipedia for word sense disambiguation has been a hot topic for some years. [1]

- The article title implies that this is somehow a spectacular finding. Doing word sense disambiguation for one word is not that interesting, and there is no comparison with existing methods to show that this is actually a high score. I suspect that it is not that spectacular, since 'Apple' is relatively easy to disambiguate using a few context words.

[1] E.g. see:

- Using Wikipedia for Automatic Word Sense Disambiguation, R. Mihalcea, 2007, for a discussion of using Wikipedia to train a word sense disambiguator.

- Integrating multiple knowledge sources to disambiguate word sense: An exemplar-based approach, H.T. Ng and H.B. Lee, 1996, provide a good overview of types of features that can be used in disambiguation. They use features that go beyond simple 'bag of word' and 'bag of n-gram' features, e.g. by using syntactical patterns.

There is a whole lot more research of course, but just to show two examples that describe far more sophisticated approaches.

Re: Python NLTK Bayesian Classifier for word sense disambiguation - 92% accuracy

#22

This is an interesting exercise in building a very specific word disambiguator ('apple' the company vs 'apple' the fruit). It is a testmanet to NLTK that this can be accomplished in less than 100 lines.

Maybe apart from stemming), it's not hard to implement this in ~100 lines without NLTK.:

- In naive Bayes classification, model parameters can usually be estimated using relative frequencies in the training data.

- WordPunctTokenizer is a very simple tokenizer that makes anything matching \w+ and [^\w\s]+ a separate token.

- Extracting Bigrams from a list of tokens is trivial.

Of course, using NLTK will be very helpful in many situations, but this is hardly a testament to NLTK.

Re: Python NLTK Bayesian Classifier for word sense disambiguation - 92% accuracy

#23
post #18

Terrible choice of words to use, since the capitalization (or not) of the word carries so much information. It's hard to take this seriously given his apparent obliviousness.

It may not be the best choice of words, but capitalization plays no role in his tests as everything is transformed to lower case on line 30: http://pastebin.com/4B1xHHht

Re: Python NLTK Bayesian Classifier for word sense disambiguation - 92% accuracy

#24

I'm surprised that no one mentioned this paper that first evaluated this approach to using Wikipedia data: http://www.aaai.org/Papers/IJCAI/2007/IJCAI07-259.pdf That said, the major drawback of using Wikipedia is the size. If this approach is to be used for all words (not just Apple) then the total training corpus will be several GBs. Definitely not practical...

What's not practical about it?

GBs of data are pretty easy to handle these days

Re: Python NLTK Bayesian Classifier for word sense disambiguation - 92% accuracy

#25
post #19
post #5

92% accuracy, unfortunately, isn't good enough. Bag-of-words models perform pretty well at classification and search, and the main thing you need to improve search is to boost scores when words are close together. You might think you could improve performance by using semantically better defined features, but even 92% accuracy adds enough noise to foil your plans. It's a big problem in A.I. systems that have multiple…

92% in general would actually be really good for word sense disambiguation, but..."Apple" is a really easy choice. I'd like to see how he does with a trickier word like "right" (as in civil, vs. not wrong, vs. not left).

Yes, it is good, but not good enough for many applications. You're also left with the issue that one kind of "apple" is more common than the other kind of "apple" so the baseline accuracy of something that always assumes it's one kind of apple might be surprisingly good.

That said, text-to-speech is a system where it's important to do disambiguation of a particular set of words. For instead,

"I read the news today, oh boy", "read" sounds like "red"

"I read the news every day", "read" sounds like "reed"

You need to be able to disambiguate the word sense to be able to correctly read the world "read". There are maybe 20 or so very common words that are like this, so a modest amount of work in this area would be part of a good TTS system.

Re: Python NLTK Bayesian Classifier for word sense disambiguation - 92% accuracy

#26
post #24

I'm surprised that no one mentioned this paper that first evaluated this approach to using Wikipedia data: http://www.aaai.org/Papers/IJCAI/2007/IJCAI07-259.pdf That said, the major drawback of using Wikipedia is the size. If this approach is to be used for all words (not just Apple) then the total training corpus will be several GBs. Definitely not practical...

What's not practical about it? GBs of data are pretty easy to handle these days

GBs and TBs of data is common, not for this task. All you are doing is Word Sense Disambiguation and there are algorithms to do WSD that work with much much smaller training sets. Just don't think that the exponential increase in training data is justified...
Post reply on HN