Pattern - Web Mining Python lib • Hacker News Board

Pattern - Web Mining Python lib

#1

Post by interro » Sun, Oct 14, 2012, 12:48 PM UTC

More info here: http://www.clips.ua.ac.be/pages/pattern

Pattern - Web Mining Python lib
github.com

Re: Pattern - Web Mining Python lib

#2

Post by interro » Sun, Oct 14, 2012, 12:50 PM UTC

More info http://www.clips.ua.ac.be/pages/pattern

Re: Pattern - Web Mining Python lib

#3

Post by mkumm » Sun, Oct 14, 2012, 2:51 PM UTC

This looks pretty interesting, I will give it a go

Re: Pattern - Web Mining Python lib

#4

Post by salimmadjd » Sun, Oct 14, 2012, 5:30 PM UTC

This is awesome! Any plans to add other sites, like amazon, yelp, tripadvisor, etc!

Re: Pattern - Web Mining Python lib

#5

Post by enjo » Sun, Oct 14, 2012, 6:34 PM UTC

salimmadjd wrote: post #4

This is awesome! Any plans to add other sites, like amazon, yelp, tripadvisor, etc!

Better yet: Is there a well defined structure for other folks to add that stuff?

Re: Pattern - Web Mining Python lib

#6

Post by stevejohnson » Sun, Oct 14, 2012, 7:02 PM UTC

salimmadjd wrote: post #4

This is awesome! Any plans to add other sites, like amazon, yelp, tripadvisor, etc!

NB: screen-scraping Yelp is against the TOS and you'll get shut down pretty fast if you try it.

Re: Pattern - Web Mining Python lib

#7

Post by Terretta » Sun, Oct 14, 2012, 7:08 PM UTC

stevejohnson wrote: post #6

salimmadjd wrote: post #4

This is awesome! Any plans to add other sites, like amazon, yelp, tripadvisor, etc!

NB: screen-scraping Yelp is against the TOS and you'll get shut down pretty fast if you try it.

Exactly. Screen scrape Google search results instead, I've heard that works great. Bet your business model on it, I've heard. ;-)

Re: Pattern - Web Mining Python lib

#8

Post by knes » Sun, Oct 14, 2012, 7:11 PM UTC

I'm a big fan of data mining so I'll make sure to take it out for spin :) And from fellow belgian people, nice!

Re: Pattern - Web Mining Python lib

#9

Post by fooandbarify » Sun, Oct 14, 2012, 9:25 PM UTC

enjo wrote: post #5

salimmadjd wrote: post #4

This is awesome! Any plans to add other sites, like amazon, yelp, tripadvisor, etc!

Better yet: Is there a well defined structure for other folks to add that stuff?

I'm not one of the authors, but the code in question is all in one file: https://github.com/clips/pattern/blob/master/pattern/web/__i...

It would be fairly straightforward to add your own class.

Re: Pattern - Web Mining Python lib

#10

Post by languagehacker » Sun, Oct 14, 2012, 9:49 PM UTC

Really cool library. I'm excited to take it for a spin! I liked that there was some work done already for Wikipedia. But as a note to people who want to work with Wikipedia data, it's not very hard to abstract your stuff to work with most wikis based on the MediaWiki platform. I've added a pull request to this project that also supports using the hundreds of thousands of wikis on Wikia. ( https://github.com/clips/pattern/pull/17 )