Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Natural Language Processing with Python, free online

nltk.org

Post reply on HN
Print view View original

21–30 of 35 posts

Re: Natural Language Processing with Python, free online

#21

Post by jnbiche »

nailer wrote: post #18
jnbiche wrote: post #13

Earlier quoted context omitted.

I'm not sure that any book on the NLTK, even one written specifically for developers, could avoid the concepts you're referring to. These terms and concepts are inherent to the study of natural language processing. So while I agree that there are a significant number of NLP-related terms used, I think that has less to do with the book's audience and more to do with the book's subject matter. And the use of computatio…

> I'm not sure that any book on the NLTK could avoid the concepts you're referring to. Me too. I'm saying the book (like almost all technical documentation) should be task-based. Let people read the chapter with their problem - don't ask them to read a whole list of solutions to work out which one is applicable. Eg: - Title: Preparing Text for Analysis (explain what tokenizing is and why it's necessary) - Title: 'gau…

Thanks for your response. Can you give me an example of how you would frame the material in your response using task-based documentation? It seems to me that the NLTK book does exactly what you're describing in most sections, but perhaps if you gave a counterexample I would better understand.

Re: Natural Language Processing with Python, free online

#22

Post by abhaga »

jnbiche wrote: post #13
nailer wrote: post #10

I own the paper version of this - just a heads up, it's written primary for Language Scientists rather than developers. You may find yourself constantly pausing and looking up concepts and their meanings / practical uses on linguistics websites when returning to the book. It's a great resource, but don't expect to get started quickly.

I'm not sure that any book on the NLTK, even one written specifically for developers, could avoid the concepts you're referring to. These terms and concepts are inherent to the study of natural language processing. So while I agree that there are a significant number of NLP-related terms used, I think that has less to do with the book's audience and more to do with the book's subject matter. And the use of computatio…

> NLTK is very well-documented and easy to work with.

Agreed.

> It's also amazingly complete -- tokenizers, stemmers, POS taggers, classifers, etc. etc etc.

Don't agree. The biggest missing piece is a statistical parser which forms the basis for a lot of further linguistic analysis. It is hard to beat Stanford Parser for that. Check out https://github.com/wavii/pfp which has Python bindings.

For most of the ML stuff, you would be better off going to a specialist library like Scikits.learn directly. They are faster and implementations are more accurate. ( I found some of the implementations not quite correct in NLTK. For example, Naive Bays classifier which a lot of first time users use. The difference in results may not be much in practice but it is still incorrect.)

It is definitely a very good place to start but better alternatives exist for many of the pieces.

Re: Natural Language Processing with Python, free online

#23

Post by jnbiche »

abhaga wrote: post #22
jnbiche wrote: post #13

Earlier quoted context omitted.

I'm not sure that any book on the NLTK, even one written specifically for developers, could avoid the concepts you're referring to. These terms and concepts are inherent to the study of natural language processing. So while I agree that there are a significant number of NLP-related terms used, I think that has less to do with the book's audience and more to do with the book's subject matter. And the use of computatio…

> NLTK is very well-documented and easy to work with. Agreed. > It's also amazingly complete -- tokenizers, stemmers, POS taggers, classifers, etc. etc etc. Don't agree. The biggest missing piece is a statistical parser which forms the basis for a lot of further linguistic analysis. It is hard to beat Stanford Parser for that. Check out https://github.com/wavii/pfp which has Python bindings. For most of the ML stuff,…

>Don't agree. The biggest missing piece is a statistical >parser which forms the basis for a lot of further linguistic >analysis.

Conceded and agreed. This is the one major gap. But I still maintain it's a remarkably complete toolkit. Plus you get to work in Python, which is a big advantage for me.

What's wrong with the Naive Bayes classifier? Did you submit a patch?

Likewise, I totally agree with you that there are faster/more accurate/more efficient implementations of many of the tools in the NLTK. If performance is a must, then you're better of prototyping in NLTK then using a specialized library. But in terms of completeness and ease of use, NLTK is very strong.

EDIT: I'm not sure why abhaga is being downvoted. There was nothing disrespectful in his response to me. Disagreement is an important part of intelligent discussion. Upvoting to counter the downvote(s).

Re: Natural Language Processing with Python, free online

#24

Post by capttwinky »

ipython+nltk+networkx+pytables+numpy+matplotlib=full of win

Seriously - text mining made fun and exploratory with open source tools.

One of the text sources I like to use is the Launchpad tickets for the Ubuntu project, since they get a good amount of traffic from international end users, a professional interest of mine.

It would be great to hear about some other interesting open data sets that people have found.

Re: Natural Language Processing with Python, free online

#25

Post by abhaga »

jnbiche wrote: post #23
abhaga wrote: post #22

Earlier quoted context omitted.

> NLTK is very well-documented and easy to work with. Agreed. > It's also amazingly complete -- tokenizers, stemmers, POS taggers, classifers, etc. etc etc. Don't agree. The biggest missing piece is a statistical parser which forms the basis for a lot of further linguistic analysis. It is hard to beat Stanford Parser for that. Check out https://github.com/wavii/pfp which has Python bindings. For most of the ML stuff,…

>Don't agree. The biggest missing piece is a statistical >parser which forms the basis for a lot of further linguistic >analysis. Conceded and agreed. This is the one major gap. But I still maintain it's a remarkably complete toolkit. Plus you get to work in Python, which is a big advantage for me. What's wrong with the Naive Bayes classifier? Did you submit a patch? Likewise, I totally agree with you that there are…

> What's wrong with the Naive Bayes classifier?

The problem I found is that it mixes up the binomial and the multinomial event models for the naive bayes (See http://www.cs.cmu.edu/~knigam/papers/multinomial-aaaiws98.pd... for reference). It computes the probabilities as per the binomial event model but doesn't include the probabilities of missing events. This was my understanding from reading the source code.

> Plus you get to work in Python, which is a big advantage for me.

Indeed. I so wish someone would build a dependency parser on top of pfp so that I can ditch Stanford parser. I have used https://github.com/dasmith/stanford-corenlp-python for interfacing with Stanford toolkit but it is somewhat brittle.

Re: Natural Language Processing with Python, free online

#26

Post by daviddaviddavid »

scorpion032 wrote: post #3

LinkGrammar: http://www.link.cs.cmu.edu/link/ is the other NLP tool kit in python, may be suited for use by those who might be intimidated by the need to write their own grammar. It is integrated into and maintained by the authors of Abiword. There is a talk on this other library at Pycon, this year: https://us.pycon.org/2012/schedule/presentation/187/

LinkGrammar is not really a toolkit and it is not written in Python. It is written in C.

Re: Natural Language Processing with Python, free online

#27

Post by ethanpoole »

jnbiche wrote: post #15
ethanpoole wrote: post #11

If you're interested in natural language processing (NLP), but don't have a linguistics background, I would suggest reading Steven Pinker's The Language Instinct. It will introduce you to the necessary terminology and concepts for NLP in an easy-to-digest way. (The NLTK book has been free online for quite some time as well.)

The Language Instinct is a great book, but unless the content of newer editions has changed significantly, it's more of an overview of linguistics in general, and language acquisition in particular. There's not much -- if any -- practical NLP. For example, looking at Amazon's statistically-improbable phrases (SIPs), I see nothing related to NLP, nor do I see any terms related to practical NLP during a quick glance of…

Oh no, I didn't intend for The Language Instinct as an introduction to NLP, but as a basic introduction to the fundamentals of linguistics. I mainly had in mind terms like morpheme, phoneme, scope, etc. A basic understanding of these concepts will make reading the NLTK book much easier, although it isn't necessary.

Re: Natural Language Processing with Python, free online

#29

Post by queensnake »

abhaga wrote: post #22
jnbiche wrote: post #13

Earlier quoted context omitted.

I'm not sure that any book on the NLTK, even one written specifically for developers, could avoid the concepts you're referring to. These terms and concepts are inherent to the study of natural language processing. So while I agree that there are a significant number of NLP-related terms used, I think that has less to do with the book's audience and more to do with the book's subject matter. And the use of computatio…

> NLTK is very well-documented and easy to work with. Agreed. > It's also amazingly complete -- tokenizers, stemmers, POS taggers, classifers, etc. etc etc. Don't agree. The biggest missing piece is a statistical parser which forms the basis for a lot of further linguistic analysis. It is hard to beat Stanford Parser for that. Check out https://github.com/wavii/pfp which has Python bindings. For most of the ML stuff,…

No SVM support either. I could try to add it I guess; libSVM has Python bindings already.

Re: Natural Language Processing with Python, free online

#30

Post by jnbiche »

queensnake wrote: post #29
abhaga wrote: post #22

Earlier quoted context omitted.

> NLTK is very well-documented and easy to work with. Agreed. > It's also amazingly complete -- tokenizers, stemmers, POS taggers, classifers, etc. etc etc. Don't agree. The biggest missing piece is a statistical parser which forms the basis for a lot of further linguistic analysis. It is hard to beat Stanford Parser for that. Check out https://github.com/wavii/pfp which has Python bindings. For most of the ML stuff,…

No SVM support either. I could try to add it I guess; libSVM has Python bindings already.

As far as I know, NLTK has no C dependencies other than its general dependency on NumPy. I think they are keeping the toolkit in pure Python on purpose (but I may be wrong about that). That said there are SVM implementations in pure Python -- PyMVPA, for one.
Post reply on HN