Really fantastic piece of software. It's about time we move away from using java wrappers for NLP stuff. Anyone know a similar project in Ruby?
Pattern: A web mining & natural language processing system for Python
21–25 of 25 posts
Re: Pattern: A web mining & natural language processing system for Python
#22Looks nice, though less full-featured than the NLTK. I'd be interested to see how nice they'd play together and whether applications could exploit the strengths of both at the same time. The only thing that's better than one good NLP framework, is two good NLP frameworks, after all.
A recurring meme in terms of frameworks is people keep bringing out tools and some of them disappear, some find niche applications, and some become mainstream. Though I haven't heard of many nlp toolkits (but I'm not in that field). I want to jump into some basic NLP, but I'd like to stick with one or two toolkits. I had heard of nltk before this, but are there any other comprehensive or sort of succesful frameworks…
The best toolkits are probably in Java:
-Stanford's Tagger, Parser, and NLP Core
-Apache OpenNLP
-Lingpipe
Many smaller components are made to be compatible with IBM UIMA (of Watson fame), so they are able to be integrated into a pipeline somewhat easily. For examples of this in biomedical TM, see http://u-compare.org/ .
People will kill me for saying this, but truly: Python's performance isn't adequate for large-scale text mining, _especially_ if you want to do deep/full parsing. Shallow parsing as shown in this package's demo is more feasible.
I personally find NLTK convoluted, but in its favor, it does have readers for a TON of corpora, which is really nice.
Re: Pattern: A web mining & natural language processing system for Python
#23Earlier quoted context omitted.
A recurring meme in terms of frameworks is people keep bringing out tools and some of them disappear, some find niche applications, and some become mainstream. Though I haven't heard of many nlp toolkits (but I'm not in that field). I want to jump into some basic NLP, but I'd like to stick with one or two toolkits. I had heard of nltk before this, but are there any other comprehensive or sort of succesful frameworks…
The best toolkits are probably in Java: -Stanford's Tagger, Parser, and NLP Core -Apache OpenNLP -Lingpipe Many smaller components are made to be compatible with IBM UIMA (of Watson fame), so they are able to be integrated into a pipeline somewhat easily. For examples of this in biomedical TM, see http://u-compare.org/ . People will kill me for saying this, but truly: Python's performance isn't adequate for large-sca…
My friends in the natural language field tell me Python and NLTK are more common than Java. Then again, this is at a sort-of Python-centric university (Toronto).
Re: Pattern: A web mining & natural language processing system for Python
#24This is all really cool stuff, but I can't help but thinking they've got a dozen or so separate packages here. They've also re-invented the wheel at every turn. All new wrapper for the Twitter API and every search engine? All new graphing library for JavaScript. Don't get me wrong, this is all awesome, but a lot of it could have been re-used from better sources without having to spend time working on random API wrapp…
The reason is mainly 1) licensing and 2) integration. I took what I found that fitted the BSD license and wrote the rest myself. The idea was having all the things I needed in a single, concise package. I like carrying around 1 MacGyver knife instead of a heavy toolbox - even if all the separate tools in the box are more robust than the knife.
As for the JavaScript graph library: Daniel Friesen had already ported 90% of the Python code, so it only took me a day or two to finish it. The result is a single file (graph.js) with lots of things besides visualization (eigenvector centrality etc.) which seemed better suited to Pattern than integrating another, bigger project.
Best, Tom
Re: Pattern: A web mining & natural language processing system for Python
#25I'm going to be involved in teaching an NLP course this semester, and we're debating what to put in it. What are some things you want to do with NLP, and what would you hope to learn (or have a future employee learn) in an honours and masters level course?