I actually just literally drooled on the keyboard!
[deleted]
Pattern: A web mining & natural language processing system for Python
11–20 of 25 posts
Re: Pattern: A web mining & natural language processing system for Python
#12Imagine spidering twitter for phrases of the type "x is a type of y" in order to form a database of real world objects in an inheritance hierarchy. Now imagine when you have these objects, finding out what these objects do by looking at verbs that occur around them. Boom. You have objects, and you have the methods you need to write. Now you just need someone to write the code! The methods writing could become a sort of captcha exercise.
Re: Pattern: A web mining & natural language processing system for Python
#13Imagine spidering twitter for phrases of the type "x is a type of y" in order to form a database of real world objects in an inheritance hierarchy. Now imagine when you have these objects, finding out what these objects do by looking at verbs that occur around them. Boom. You have objects, and you have the methods you need to write. Now you just need someone to write the code! The methods writing could become a sort…
All of this is to upload the universe of course into some sort of Minecraft game.
Re: Pattern: A web mining & natural language processing system for Python
#14Project thats been in the back of my head for a while, but have no time to do: Analyze HN comments over time with some NLP techniques, maybe sentiment analysis. Then if the next wave of "HN is turning into Reddit" posts comes, point people to the analysis, whatever the conclusions are. Seems like this would be well suited for the task. Any takers?
Use our API:
Re: Pattern: A web mining & natural language processing system for Python
#15This is all really cool stuff, but I can't help but thinking they've got a dozen or so separate packages here. They've also re-invented the wheel at every turn. All new wrapper for the Twitter API and every search engine? All new graphing library for JavaScript.
Don't get me wrong, this is all awesome, but a lot of it could have been re-used from better sources without having to spend time working on random API wrappers and the like. I would definitely like to know the reasoning behind creating everything from scratch.
Re: Pattern: A web mining & natural language processing system for Python
#16Imagine spidering twitter for phrases of the type "x is a type of y" in order to form a database of real world objects in an inheritance hierarchy. Now imagine when you have these objects, finding out what these objects do by looking at verbs that occur around them. Boom. You have objects, and you have the methods you need to write. Now you just need someone to write the code! The methods writing could become a sort…
i.e. the semantic web, circa forever?
Re: Pattern: A web mining & natural language processing system for Python
#17I don't know a whole lot about text analysis and the mentioned algorithms, can this be used to analyze articles and determine which are dealing with the same subject? Techmeme-ish? Or what would be a good starting point for this? (Or would this be better off in an 'Ask HN' post? I am one of those horrible new people on here.)
The: "tf-idf + cosine similarity + LSA metrics" bit from Pattern is what you are looking for.
In other words, the vector module: http://www.clips.ua.ac.be/pages/pattern-vector
Re: Pattern: A web mining & natural language processing system for Python
#18Imagine spidering twitter for phrases of the type "x is a type of y" in order to form a database of real world objects in an inheritance hierarchy. Now imagine when you have these objects, finding out what these objects do by looking at verbs that occur around them. Boom. You have objects, and you have the methods you need to write. Now you just need someone to write the code! The methods writing could become a sort…
You need to check out Freebase.
Re: Pattern: A web mining & natural language processing system for Python
#19This is all really cool stuff, but I can't help but thinking they've got a dozen or so separate packages here. They've also re-invented the wheel at every turn. All new wrapper for the Twitter API and every search engine? All new graphing library for JavaScript. Don't get me wrong, this is all awesome, but a lot of it could have been re-used from better sources without having to spend time working on random API wrapp…
May be they wanted it to fit together more seamlessly than if they were to use the existing pieces?
Re: Pattern: A web mining & natural language processing system for Python
#20Looks nice, though less full-featured than the NLTK. I'd be interested to see how nice they'd play together and whether applications could exploit the strengths of both at the same time. The only thing that's better than one good NLP framework, is two good NLP frameworks, after all.
A recurring meme in terms of frameworks is people keep bringing out tools and some of them disappear, some find niche applications, and some become mainstream. Though I haven't heard of many nlp toolkits (but I'm not in that field).
I want to jump into some basic NLP, but I'd like to stick with one or two toolkits. I had heard of nltk before this, but are there any other comprehensive or sort of succesful frameworks out there one should be aware of? (Either in python or something else)